Aircraft intelligent maneuvering penetration decision-making method based on double-layer deep reinforcement learning

Abstract

The invention particularly relates to an aircraft intelligent maneuvering penetration decision method based on double-layer deep reinforcement learning, and the method comprises the steps: constructing first self observation state data according to the flight state parameters of an autonomous aircraft; constructing second self observation state data according to the flight state parameters of the interception equipment; third state space parameters are constructed according to the relative state parameters between the autonomous aircraft and the interception equipment; on the basis of the first self-observation state data, the second self-observation state data and the third state space parameter, constructing overall state space data; configuring the overall state space data as input parameters of an upper-layer decision network, and performing reinforcement learning on the input parameters by using the upper-layer decision network to obtain a penetration decision judgment result output by the network; wherein the penetration decision judgment result is used for determining whether the current autonomous aircraft needs to execute maneuvering penetration. According to the method, adaptive intelligent decisions can be made for defense penetration opportunities aiming at real-time changing environments.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to an intelligent maneuver penetration decision-making method for an aircraft based on double-layer deep reinforcement learning. Background Art

[0002] In related technologies, aircraft maneuver penetration strategies can be divided into two categories: programmed maneuver penetration and intelligent maneuver penetration. In traditional programmed maneuver penetration strategies, common programmed maneuvers include sine maneuvers, square wave maneuvers, and spiral maneuvers. Such maneuvering methods can achieve a large miss distance through certain parameter combinations, trajectory combinations, or guidance law designs to improve the penetration ability; however, the limitations of such methods are also very obvious. The aircraft flies programmatically based on the situational information in the known conflict area, which is obtained during the pre-launch planning process of the aircraft and has a strong dependence on humans and external information support. However, due to the continuous change of the environment in the conflict area and the incomplete situational information in the conflict area, the traditional penetration strategy based on a preset flight program fails, and the correlation between the penetration time and the penetration method is ignored, making it impossible to cope with multiple interception devices simultaneously.

[0003] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0004] The present invention provides an intelligent maneuver penetration decision-making method for an aircraft based on double-layer deep reinforcement learning and a computer program product, which can overcome the defects existing in the prior art to a certain extent.

[0005] Other features and advantages of the present invention will become apparent through the following detailed description, or will be partially learned through the practice of the present invention.

[0006] According to a first aspect of the present invention, there is provided an intelligent maneuver penetration decision-making method for an aircraft based on double-layer deep reinforcement learning, the method comprising:

[0007] Constructing first self-observation state data according to the flight state parameters of the autonomous aircraft;

[0008] Constructing second self-observation state data according to the flight state parameters of the interception device;

[0009] Constructing third state space parameters according to the relative state parameters between the autonomous aircraft and the interception device;

[0010] Constructing overall state space data based on the first self-observation state data, the second self-observation state data, and the third state space parameters;

[0011] Configure the overall state space data as the input parameter of the upper - layer decision network, and use the upper - layer decision network to perform reinforcement learning on the input parameter to obtain the penetration decision - making judgment result output by the network; wherein, the penetration decision - making judgment result is used to determine whether the current autonomous aircraft needs to perform maneuvering penetration.

[0012] In some exemplary embodiments, the method further includes:

[0013] Construct the fourth state space data according to the relative state parameters between the autonomous aircraft and the target, so as to construct the overall state space data in combination with the fourth state space data.

[0014] In some exemplary embodiments, the relative state parameters between the autonomous aircraft and the target include any one or any combination of the following: the relative distance between the autonomous aircraft and the target, the line - of - sight elevation angular velocity, the line - of - sight direction angular velocity, the line - of - sight elevation angle, and the line - of - sight direction angle.

[0015] In some exemplary embodiments, the method further includes:

[0016] Configure the constraint conditions corresponding to the fourth state space data; the constraint conditions include at least one of the trajectory deviation degree of the autonomous aircraft and the flight altitude of the autonomous aircraft.

[0017] In some exemplary embodiments, the first self - observation state data includes the ballistic inclination angle parameter and the ballistic deflection angle parameter of the autonomous aircraft;

[0018] The second self - observation state data includes the ballistic inclination angle parameter and the ballistic deflection angle parameter of the interception equipment.

[0019] In some exemplary embodiments, the third state space parameters include any one or any combination of the following between the autonomous aircraft and the interception equipment: relative distance, line - of - sight elevation angular velocity, line - of - sight direction angular velocity, line - of - sight elevation angle, and line - of - sight direction angle.

[0020] In some exemplary embodiments,

[0021] When the penetration decision - making judgment result output by the upper - layer decision network is that the autonomous aircraft currently needs to perform maneuvering penetration, the method further includes:

[0022] Configure the overall state space data as the input parameter of the lower - layer decision network, and use the lower - layer decision network to process the input parameter to obtain the penetration action data output by the network; wherein, the penetration action data includes the normal acceleration and / or normal overload of the autonomous aircraft.

[0023] In some exemplary embodiments, the method further includes:

[0024] When it is recognized that there is intercept equipment for the autonomous aircraft, calculate the real-time distance between the autonomous aircraft and the intercept equipment;

[0025] When the real-time distance is less than a preset distance threshold, trigger the maneuvering penetration mission of the autonomous aircraft;

[0026] Execute the maneuvering penetration mission of the autonomous aircraft to construct the overall state space data and determine whether the current autonomous aircraft needs to execute the maneuvering penetration.

[0027] In some exemplary embodiments, the method further includes: pre-training an autonomous aircraft maneuvering penetration decision model, including:

[0028] Initialize the basic task parameters, including: network basic parameters, guidance ratio, and penetration reward;

[0029] Configure the motion relationship between the autonomous aircraft, the intercept equipment, and the target;

[0030] Construct sample overall state space data according to the sample state parameters of the autonomous aircraft and the intercept equipment;

[0031] Input the sample overall state space data into the upper-level decision network to obtain the sample penetration decision judgment result output by the network; wherein, the upper-level decision network is constructed based on the DDQN model;

[0032] Input the sample overall state space data into the lower-level decision network to obtain the sample penetration action data output by the network; wherein the lower-level decision network is constructed based on the TD3 model;

[0033] The autonomous aircraft executes the sample penetration action data to obtain optimized trajectory data;

[0034] Calculate the reduction of ballistic deviation according to the optimized trajectory data, and calculate the energy loss reward according to the penetration action data; repeat the above steps until the autonomous aircraft breaks through the intercept equipment, or is intercepted by the intercept equipment or is lower than the flight altitude constraint of the autonomous aircraft;

[0035] Evaluate the penetration effect according to the penetration result of the autonomous aircraft to obtain the penetration evaluation result; wherein, the rewards of the upper-level decision network include at least one of the flight lower bound limit reward, the penetration determination reward, and the energy loss reward; the rewards of the lower-level decision network include at least one of the flight lower bound limit reward and the penetration determination reward;

[0036] Construct sample data according to the sample overall state space data, the sample penetration decision judgment result, the sample penetration action data, and the penetration evaluation result, and save it to the experience pool;

[0037] Repeat the above steps to iteratively train the autonomous aircraft maneuvering penetration decision-making model.

[0038] According to a second aspect of the present invention, there is provided a computer program product, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned aircraft intelligent maneuvering penetration decision-making method based on double-layer deep reinforcement learning is implemented.

[0039] According to a third aspect of the present invention, there is provided an electronic device, including:

[0040] a processor; and

[0041] a memory for storing executable instructions of the processor; wherein, the memory is used for storing the executable instructions of the processor; the processor is configured to implement the above-mentioned aircraft intelligent maneuvering penetration decision-making method based on double-layer deep reinforcement learning when executing the executable instructions.

[0042] According to a fourth aspect of the present invention, there is provided a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned aircraft intelligent maneuvering penetration decision-making method based on double-layer deep reinforcement learning is implemented.

[0043] The aircraft intelligent maneuvering penetration decision-making method based on double-layer deep reinforcement learning provided by the embodiments of the present invention can accurately represent the real-time situation between the autonomous aircraft and the interception equipment by using the flight state parameters and relative state parameters of the autonomous aircraft and the interception equipment to construct the overall state space data as the observation matrix. By using the overall state space data as the input of the upper-layer decision-making network, it is possible to accurately judge whether the autonomous aircraft needs to perform maneuvering penetration against the emerging interception equipment at present, realize an adaptive intelligent decision on the penetration opportunity for the real-time changing environment, and thus ensure the effect of the penetration decision. Furthermore, it can ensure the effective completion of the subsequent tasks of the autonomous aircraft.

[0044] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0046] Figure 1Schematic diagram showing a method for intelligent maneuvering penetration decision-making of an aircraft based on double-layer deep reinforcement learning according to an exemplary embodiment of the present invention;

[0047] Figure 2 Schematic diagram showing the three-dimensional motion relationship of a penetration scenario according to an exemplary embodiment of the present invention;

[0048] Figure 3 Schematic diagram showing a model training method according to an exemplary embodiment of the present invention;

[0049] Figure 4 Schematic diagram showing an upper-layer training reward curve according to an exemplary embodiment of the present invention;

[0050] Figure 5 Schematic diagram showing a lower-layer training reward curve according to an exemplary embodiment of the present invention;

[0051] Figure 6 Schematic diagram showing the three-dimensional trajectory of multi-interceptor interception offense and defense confrontation according to an exemplary embodiment of the present invention;

[0052] Figure 7 Schematic diagram showing an enlarged view of the end of the three-dimensional trajectory of multi-interceptor interception offense and defense confrontation according to an exemplary embodiment of the present invention;

[0053] Figure 8 Schematic diagram showing the Y-direction control command of an offensive missile according to an exemplary embodiment of the present invention;

[0054] Figure 9 Schematic diagram showing the Z-direction control command of an offensive missile according to an exemplary embodiment of the present invention;

[0055] Figure 10 Schematic diagram showing the curve of the relative distance change between an interceptor and an offensive missile according to an exemplary embodiment of the present invention;

[0056] Figure 11 Schematic diagram showing an electronic device according to an exemplary embodiment of the present invention. Detailed implementation manners

[0057] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this invention will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0058] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0059] In the related art, the existing programmatic maneuvering penetration scheme highly depends on prior battlefield rules; in addition, the existing maneuvering penetration scheme cannot be specifically changed according to the real environment in the battlefield.

[0060] In view of the disadvantages and deficiencies of the prior art, in the present exemplary embodiment, an intelligent maneuvering penetration decision-making method for an aircraft based on double-layer deep reinforcement learning is provided, which can improve the effectiveness in dealing with multiple interception devices. Referring to Figure 1 as shown, the method may include the following steps:

[0061] Step S11, constructing first self-observation state data according to the flight state parameters of the autonomous aircraft;

[0062] Step S12, constructing second self-observation state data according to the flight state parameters of the interception device;

[0063] Step S13, constructing third state space parameters according to the relative state parameters between the autonomous aircraft and the interception device;

[0064] Step S14, constructing overall state space data based on the first self-observation state data, the second self-observation state data and the third state space parameters;

[0065] Step S15, configuring the overall state space data as input parameters of an upper-layer decision-making network, and performing reinforcement learning on the input parameters by using the upper-layer decision-making network to obtain a penetration decision-making judgment result output by the network; wherein, the penetration decision-making judgment result is used to determine whether the current autonomous aircraft needs to perform maneuvering penetration.

[0066] Next, each step of the intelligent maneuvering penetration decision-making method for an aircraft based on double-layer deep reinforcement learning in the present exemplary embodiment will be described in more detail with reference to the accompanying drawings and embodiments. [[ID=XX]] [[ID=XX]]

[0067] Exemplarily, the above-mentioned autonomous aircraft may be a flight device such as a drone, a missile, a highly maneuverable aircraft, etc. The interception device may be a drone, a missile for realizing the interception function, or a flight device launched by a defense system.

[0068] Exemplarily, the method further includes:

[0069] Step S101, when it is recognized that there is an interception device targeting the autonomous aircraft, calculate the real-time distance between the autonomous aircraft and the interception device;

[0070] Step S102, when the real-time distance is less than a preset distance threshold, trigger the maneuvering penetration mission of the autonomous aircraft;

[0071] Step S103, execute the maneuvering penetration mission of the autonomous aircraft to construct the overall state space data and determine whether the current autonomous aircraft needs to execute the maneuvering penetration.

[0072] Specifically, during the process of the autonomous aircraft flying towards the predetermined target according to the configured path, it can be recognized in real time whether there is an interception device targeting the autonomous aircraft. For example, it can be detected whether there is an interception device currently through radar equipment. When it is recognized that there is currently an interception device targeting the autonomous aircraft, data such as the flight speed, direction, and altitude of the interception device can be detected. And the real-time distance between the autonomous aircraft and the interception device can be calculated in real time. If the real-time distance is less than the preset distance threshold, the maneuvering penetration mission of the autonomous aircraft can be triggered on the terminal device side or on the autonomous aircraft side. When executing the maneuvering penetration mission of the autonomous aircraft, various parameters of the autonomous aircraft and the interception device can be collected. Taking the interception device as a missile as an example, the ballistic inclination angle and ballistic deflection angle of the missile can also be detected and calculated through radar equipment.

[0073] In step S11, according to the flight state parameters of the autonomous aircraft, construct the first self-observation state data.

[0074] Exemplarily, the first self-observation state data includes the ballistic inclination angle parameter and ballistic deflection angle parameter of the autonomous aircraft.

[0075] In step S12, according to the flight state parameters of the interception device, construct the second self-observation state data.

[0076] Exemplarily, the second self-observation state data includes the ballistic inclination angle parameter and ballistic deflection angle parameter of the interception device.

[0077] Specifically, when making a decision on the maneuvering penetration of the autonomous aircraft, the flight state parameters of the autonomous aircraft and the interception device can be obtained first and used to construct the first self-observation state data and the second self-state data. Among them, there can be at least one interception device. For example, if the radar equipment currently detects one, two, or three interception devices, the corresponding second self-observation state data can be constructed for each interception device respectively.

[0078] Specifically, to enable the decision-making network to have the ability to "act according to its capabilities", that is, to evaluate its penetration ability against multiple interception equipment considering its own penetration ability, the first thing to consider is the own state information of the autonomous vehicle and the interception equipment. The following state formula is used in the state space:

[0079]

[0080] Among them, and O

[0085] respectively represent the state spaces of the interception equipment and the autonomous vehicle; "D j " represents the own observation state of the interception equipment j; "M" represents the own observation state of the autonomous vehicle. Taking the autonomous vehicle and the interception equipment as missiles as an example, θ represents the ballistic inclination angle of the missile, and ψ v represents the ballistic deflection angle of the missile.

[0081] In step S13, according to the relative state parameters between the autonomous vehicle and the interception equipment, the third state space parameters are constructed.

[0082] Exemplarily, the third state space parameters include any one or any combination of the following between the autonomous vehicle and the interception equipment: relative distance, line-of-sight elevation angular velocity, line-of-sight direction angular velocity, line-of-sight elevation angle, and line-of-sight direction angle.

[0083] Specifically, during the penetration process, in addition to considering the own state information of the autonomous vehicle and the interception equipment, the relative information between the autonomous vehicle and the interception equipment should also be considered for active avoidance. In the mission process analysis, the line-of-sight angle is easier to obtain than the distance and reflects the threat level of the target to a certain extent. Therefore, the relative line-of-sight angle is considered as one of the subjective observation information. The following state formula is used in the state space:

[0084]

[0085] Among them, represents the relative state observation space between the attacking missile and the interception equipment j, that is, the third state space parameters; "MD j " represents the relative information between the autonomous vehicle and the interception equipment j, and R represents the relative distance; and respectively represent the line-of-sight elevation angular velocity and the line-of-sight direction angular velocity; q ε and q β respectively represent the line-of-sight elevation angle and the line-of-sight direction angle.

[0086] In step S14, based on the first own observation state data, the second own observation state data, and the third state space parameters, the overall state space data is constructed.

[0087] Specifically, after obtaining the above-mentioned various data, the above-mentioned various observables can be spliced to obtain an observation data matrix as the overall state space data. It can be specifically expressed as:

[0088]

[0089] Among them, O represents the overall state space.

[0090] Preferably, in some exemplary embodiments, the method further includes: constructing fourth state space data according to the relative state parameters between the autonomous vehicle and the target, so as to construct the overall state space data in combination with the fourth state space data.

[0091] Exemplarily, the relative state parameters between the autonomous vehicle and the target include: any one or any combination of the relative distance between the autonomous vehicle and the target, the line-of-sight elevation angular velocity, the line-of-sight direction angular velocity, the line-of-sight elevation angle, and the line-of-sight direction angle.

[0092] Exemplarily, the method further includes:

[0093] Configuring the constraint conditions corresponding to the fourth state space data; the constraint conditions include at least one of the trajectory deviation degree of the autonomous vehicle and the flight altitude of the autonomous vehicle.

[0094] Specifically, referring to Figure 2 As shown, taking the current situation including one interception device as an example, the motion relationship between the autonomous vehicle, the target and the interception device can be constructed in the coordinate system. In order to prevent the autonomous vehicle from ignoring subsequent tasks due to simple penetration, the trajectory deviation degree should be considered according to the relative information of the target during the penetration process, as well as the flight lower limit (minimum altitude) during the flight. Among them, taking the autonomous vehicle as a missile as an example, the trajectory deviation degree can be the ballistic deviation degree. The following state formula is used in the state space:

[0095]

[0096] r_MV represents the relative state space between the autonomous vehicle and the target, that is, the fourth state space data. "MV" represents the relative information between the autonomous vehicle and the target.

[0097] Based on the first self-observation state data, the second self-observation state data, the third state space parameter, and the fourth state space data, the overall state space data can be constructed. By splicing all the above-mentioned observables, the final state is:

[0098]

[0099] ​Among them, O represents the overall state space.

[0100] In some exemplary embodiments, the above parameters can be obtained asynchronously; alternatively, multiple data acquisition tasks can also be executed simultaneously, so as to synchronously obtain the parameters among the autonomous aircraft, the interception equipment, and the target.

[0101] In step S14, the overall state space data is configured as the input parameter of the upper decision-making network, and the upper decision-making network performs reinforcement learning on the input parameter to obtain the penetration decision judgment result output by the network; among them, the penetration decision judgment result is used to determine whether the current autonomous aircraft needs to perform maneuver penetration.

[0102] Exemplarily, after obtaining the overall state space data, this data is used to represent the current situation between the autonomous aircraft and the interception equipment. It can be input into the autonomous aircraft maneuver penetration decision model. Among them, the autonomous aircraft maneuver penetration decision model includes: an upper decision-making network and a lower decision-making network. Among them, the upper decision-making network is constructed based on the DDQN network (double deep Q network, double-layer deep Q network). The overall state space data is input into the agent, and the upper decision-making network is used to obtain the output action, and the formula can be expressed as:

[0103] A DDQN =[a DDQN

[0104] Among them, when A DDQN =[0], it means that the current interception equipment will not affect the flight trajectory of the autonomous aircraft, and the autonomous aircraft continues to fly along the original flight route. Or, when A DDQN =[1], it means that it is predicted that the interception equipment will affect the flight trajectory of the autonomous aircraft. At this time, it is determined that the current autonomous aircraft needs to perform maneuver penetration, and the autonomous aircraft can start to calculate the maneuver penetration decision.

[0105] Exemplarily, the method further includes:

[0106] Step S16, when the penetration decision judgment result output by the upper decision-making network is that the current autonomous aircraft needs to perform maneuver penetration, the overall state space data is configured as the input parameter of the lower decision-making network, and the lower decision-making network processes the input parameter to obtain the penetration action data output by the network; among them, the penetration action data includes the normal acceleration and / or normal overload of the autonomous aircraft.

[0107] ​Specifically, the lower-level decision-making network can be constructed based on the TD3 network. After determining that the current autonomous aircraft needs to perform maneuvering penetration, the overall state space data can be input into the lower-level decision-making network of the autonomous aircraft's maneuvering penetration decision-making model to obtain the penetration action data output by the lower-level decision-making network. The formula can be expressed as:

[0108]

[0109] Wherein, and respectively represent the normal acceleration of the autonomous aircraft in the y direction and the normal overload in the z direction.

[0110] Specifically, this method first designs an adaptive state description based on real-time information such as changes in the motion state of the interceptor missile, changes in the motion state of the attacking missile, and changes in the relative information between the attacking missile, the interceptor missile, and the virtual target. Subsequently, the action structure of the double-layer network is designed to realize the adaptive decision-making of the missile intelligent maneuvering penetration strategy network for the penetration timing and penetration method.

[0111] Exemplarily, the method further includes: pre-training the autonomous aircraft's maneuvering penetration decision-making model, including:

[0112] Step S201, initialize the basic parameters of the task, including: network basic parameters, guidance ratio, and penetration reward;

[0113] Step S202, configure the motion relationship among the autonomous aircraft, the interception equipment, and the target;

[0114] Step S203, construct the sample overall state space data according to the sample state parameters of the autonomous aircraft and the interception equipment;

[0115] Step S204, input the sample overall state space data into the upper-level decision-making network to obtain the sample penetration decision judgment result output by the network; wherein, the upper-level decision-making network is constructed based on the DDQN model;

[0116] Step S205, input the sample overall state space data into the lower-level decision-making network to obtain the sample penetration action data output by the network; wherein the lower-level decision-making network is constructed based on the TD3 model;

[0117] Step S206, the autonomous aircraft executes the sample penetration action data to obtain the optimized trajectory data;

[0118] Step S207, calculate the reduction of the ballistic deviation according to the optimized trajectory data, and calculate the energy loss reward according to the penetration action data; repeat the above steps until the autonomous aircraft breaks through the interception equipment, or is intercepted by the interception equipment or below the flight altitude constraint of the autonomous aircraft;

[0119] Step S208, evaluate the penetration effect according to the penetration result of the autonomous aircraft to obtain a penetration evaluation result; among them, the rewards of the upper-layer decision network include at least one of the flight lower bound limit reward, the penetration determination reward, and the energy loss reward; the rewards of the lower-layer decision network include at least one of the flight lower bound limit reward and the penetration determination reward;

[0120] Step S209, construct sample data based on the overall state space data of the sample, the sample penetration decision judgment result, the sample penetration action data, and the penetration evaluation result, and save it to the experience pool;

[0121] Repeat the above steps to iteratively train the maneuver penetration decision model of the autonomous aircraft.

[0122] Specifically, the maneuver penetration decision model of the autonomous aircraft includes an upper-layer decision network constructed based on the DDQN network and a lower-layer decision network constructed based on the TD3 network. During model training, refer to Figure 3 as shown:

[0123] Step 1, first set the basic task parameters, including network basic parameters, guidance ratio, penetration reward, etc. In addition, in the simulation environment of the training process, set the number and flight speed of the autonomous aircraft and the interception equipment, etc.

[0124] Step 2, the simulation environment can be initialized. In the single autonomous aircraft maneuver penetration task under multiple interception equipment, there are three types: interception equipment, the autonomous aircraft, and virtual targets on the flight path of the original autonomous aircraft. The autonomous aircraft flies at extremely low altitude over the sea surface. After the enemy detects the autonomous aircraft, it launches interception equipment to intercept. After the autonomous aircraft detects the interception equipment, it gives a virtual target to the penetration strategy to avoid the interception equipment. Therefore, the three-dimensional motion relationship between the autonomous aircraft and the virtual target and the interception equipment is as Figure 2 shown.

[0125] Step 3, the observation matrix can be calculated. Since both the penetration timing and the penetration method are judged according to the situation of both sides, the same state space is used for observation. Here, take the autonomous aircraft as the offensive missile and the interception equipment as the interception missile as an example.

[0126] In order to enable the decision network to have the ability of "acting according to one's ability", that is, to evaluate its own penetration ability against multiple interception equipment considering its own penetration ability, the first thing to consider is the own state information of the offensive missile and the interception missile. The following state formula is used in the state space:

[0127]

[0128] Among them, and O Mrespectively represent the state spaces of the interceptor missile j and the attacking missile; the subscript "D j " represents the self-observed state of the interceptor missile j; the subscript "M" represents the self-observed state of the attacking missile; θ represents the ballistic inclination angle of the missile, and ψ v represents the ballistic deflection angle of the missile.

[0129] During the penetration process, in addition to considering the self-state information of the attacking missile and the interceptor missile, the relative information between the attacking missile and the interceptor missile should also be considered for active avoidance. In the mission process analysis, the line-of-sight angle is easier to obtain than the distance, and to a certain extent, it reflects the threat level of the target. Therefore, in this chapter, the relative line-of-sight angle is used as one of the subjective measurement information. The following state equations are used in the state space:

[0130]

[0131] where represents the relative state observation space between the attacking missile and the interceptor missile j; the subscript "MD j " represents the relative information between the missile and the interceptor missile j; R represents the relative distance; and respectively represent the line-of-sight elevation angular velocity and the line-of-sight direction angular velocity; q ε and q β respectively represent the line-of-sight elevation angle and the line-of-sight direction angle.

[0132] One cannot ignore the subsequent strike mission for the sake of simple penetration. During the penetration process, the degree of ballistic deviation and the flight lower limit (minimum altitude) during flight should be considered according to the relative information of the virtual target. The following state equations are used in the state space:

[0133]

[0134] where, O r_MV represents the relative state observation space between the attacking missile and the virtual point, and the subscript "MV" represents the relative information between the missile and the virtual point.

[0135] By splicing all the above-mentioned observed quantities, the final state is obtained as:

[0136]

[0137] where, O represents the overall state space.

[0138] Step 4, calculate the output action of the upper-level network. Specifically, judge whether the penetration starts according to the situation of both sides at all times, and input the state space into the intelligent agent DDQN network to obtain the output action:

[0139] A DDQN = [a DDQN ​

[0140] Step 5, Penetration mission simulation.

[0141] After obtaining the action output at the penetration opportunity, the simulation environment determines the simulation at the penetration moment according to the action. Among them, the interceptor uses the proportional navigation method with the guidance parameter k for guidance operation. After obtaining the action at a certain moment, the simulation environment adjusts the state of the attacking missile.

[0142] When A DDQN =[0], the attacking missile continues to fly along the original flight route, then repeats the previous step and continues the current simulation; when A DDQN =[1], start to execute the penetration mode decision.

[0143] Among them, the upper-level decision-making network constructed based on the DDQN network can include: Q network and Target Q network. Among them, given a state s t , obtain the Q values of various actions through the Q network, and then select the action a t using the greedy strategy, find the action with the maximum output value from the Q network, and then find the output value of the Target Q network corresponding to this action. Specifically, first according to s t , a t , the current Q(s t , a t ) can be determined; assume it is Q(s t , a1). Then input the next state s t+1 into the Q network to obtain the Q values of various different actions. Next, based on complete greed, select the action corresponding to the maximum Q value. Assume the action a1 is selected. Then input the state s t+1 into the TargetQ network to find the Q value Q(s t+1 , a1) corresponding to the action a1. Finally, Q(s t , a1) is used as the predicted value of the network, and r t+1 + γQ(s t+1 , a1) is used as the actual value of the network, and error backpropagation is performed. After a period of time, the parameters in the Q network are hard copied to the Target Q network.

[0144] During the model training process, when the number of samples is sufficient, sample a small batch of samples from the experience buffer, use the online network in DDQN to select the optimal action for the next state, then calculate the target Q value using the target network, calculate the loss and update the parameters of the online network, and then update the parameters of the target network every fixed deployment.

[0145] Step 6, Calculate the action output of the lower-level network.

[0146] According to the situation of both sides at every moment, we can judge the form of maneuver to break through the defense, input the current state space into the agent TD3 network to obtain the output action:

[0147]

[0148] in, and They represent the normal acceleration in the y direction and the normal overload in the z direction of the attacking projectile respectively.

[0149] Among them, the TD3 network (Twin Delayed Deep Deterministic Policy Gradient) includes an Actor-Critic architecture, in which the Actor network is responsible for generating actions and the Critic network is responsible for evaluating the value of actions (Q-value). This architecture enables the algorithm to efficiently solve high-dimensional continuous action problems. The Actor update goal is to maximize the Q-value of the Critic network, while the Critic network optimization goal is to minimize the Q-value prediction error. The TD3 network can directly generate actions by learning a deterministic policy. In addition, TD3 uses two independent Critic networks to calculate the Q-value, and takes the minimum value of the two as the target Q-value.

[0150] During model training, when the number of samples is sufficient, small batches of samples are sampled from the experience buffer, and the target actor network in TD3 is used to generate the action for the next state. Subsequently, the target critic network is used to calculate the target value, update the dual critic network, minimize the mean squared error, delay the update of the actor network, and soft-update the target network parameters.

[0151] Step 7, Penetration Mission Simulation: This is the trajectory deviation reward evaluation step of one of the designs of the present invention.

[0152] After obtaining the output of the penetration mode action, the simulation environment simulates the penetration mode decision based on the action. At this time, the environment simulates with a fixed step size.

[0153] After obtaining the action at a certain moment, the attacking missile in the simulation environment adjusts the action. The normal acceleration of the attacking missile in the y direction executes the action. The normal acceleration of the attacking projectile in the z direction executes the action

[0154] After the current simulation step is completed, a timely reward evaluation is performed. The timely reward includes the trajectory deviation reward and the energy loss reward.

[0155] The purpose of the trajectory deviation bonus is to reduce the trajectory deviation of the missile during the penetration maneuver.

[0156]

[0157] The purpose of the energy loss reward is to minimize the energy consumption of the missile as much as possible on the basis of successful penetration, which is:

[0158]

[0159] If the total reward is:

[0160] R j = R V + R loss

[0161] Return to the output action in step 6 and continue the current simulation until all interceptors are broken through or the attacking missiles are intercepted, or below the lower flight bound.

[0162] Step 8, Penetration effect evaluation.

[0163] In order to enable evasion through maneuvering actions when facing multiple missile interceptions, a terminal return reward is designed to guide the missile to achieve the penetration mission.

[0164] The terminal reward is used to measure whether the mission is successful, mainly involving whether the interceptors are successfully evaded and whether the mission is completed according to the upper and lower flight bound constraints. At the same time, in order to guide the attacking missile to learn the correct maneuvering penetration strategy, the reward for successful penetration is related to the amount of miss of the interceptor.

[0165] The reward for the lower flight bound limit is:

[0166]

[0167] The reward for penetration determination is:

[0168]

[0169] For the upper layer network, it mainly determines whether to perform a penetration action according to the current moment, and at the same time evaluates the penetration success rate and energy loss after performing the action at this moment. Therefore, the terminal reward for the upper layer network is:

[0170] R DDQN = R loss_all + R X + R limit_y

[0171] Among them, R loss_all is the sum of the energy loss rewards for each step in a round of simulation.

[0172] For the lower layer network, it mainly determines whether the penetration action is conducive to successful penetration according to the current moment. Therefore, the terminal reward for the lower layer network is:

[0173] R TD3 = Rlimit_y +R x

[0174] Step 9: Organize the "experience" obtained from this penetration mission. Organize and save the observation matrices, allocation actions, and penetration effects of the offensive missiles, interceptor missiles, and virtual targets in this mission. Through multiple mission simulations, a simulation experience pool can be obtained, providing a basis for parameter optimization in the training of the agent network. Thus, one training round or a single mission simulation of this method is completed.

[0175] Repeat the above steps to iteratively train the maneuver penetration decision-making model of the autonomous aircraft and obtain a trained decision-making model.

[0176] Specifically, by utilizing the collaborative optimization mechanism of the upper and lower layers of the double-layer framework, this method feeds back the results of the penetration method decision and the information before and after it to the upper-layer strategy, dynamically adjusts the penetration timing, and makes the timing selection and method optimization complement each other; introduces virtual targets to reduce the ballistic deviation during penetration and ensure subsequent combat missions; at the same time, this technology designs a new evaluation function to realize the self-evaluation of the penetration plan and network update through the effect evaluation of the solutions generated by each algorithm.

[0177] Exemplarily, a simulation environment where three interceptor missiles intercept an offensive missile is used as an example for illustration. Table 1 shows the parameters of the simulation environment and the network infrastructure settings. Table 2 shows the parameter settings of the upper-layer network DDQN. Table 3 shows the parameter settings of the lower-layer network TD3.

[0178] Table 1

[0179]

[0180] Table 2

[0181]

[0182] Table 3

[0183]

[0184] Both the interceptor missiles and the offensive missiles are randomly processed based on their positions and velocities within a fixed range and then dispersed to obtain the initial state of the example given this time.

[0185] Step 1:

[0186] Set the basic mission parameters, including network infrastructure parameters, guidance ratio, penetration reward, etc. In addition, in the simulation environment for the training process, set the number and flight speeds of the offensive missiles and interceptor missiles, etc.

[0187] Step 2:

[0188] Initialize the simulation environment. Obtain the initial positions and velocities of the missiles and targets in this simulation.

[0189] The information of the three interceptor missiles obtained at the beginning of a certain round is shown in Table 4 below.

[0190] Table 4

[0191]

[0192] Step 3:

[0193] Calculate the observation matrix. The observation information of the attacking missile used in this method is expressed as follows:

[0194]

[0195] It should be noted here that in order to ensure the stability of the network output, the state space has been normalized.

[0196] Step 4:

[0197] Calculate the output action of the upper-layer network.

[0198] Judge whether the penetration starts according to the situation of both sides at all times, and input the state space into the intelligent agent DDQN network to obtain the output action:

[0199] A DDQN =[a DDQN

[0200] Taking the missile initialization result in this simulation as an example, output A DDQN =[0].

[0201] Step 5:

[0202] Penetration mission simulation. After obtaining the penetration timing action output in Step 4, the simulation environment determines the penetration moment according to the action. Among them, the interceptor missile uses the proportional navigation method with the guidance parameter k for guidance operation.

[0203] After obtaining the action at a certain moment, the simulation environment adjusts the state of the attacking missile. When A DDQN =[0], the attacking missile continues to fly along the original flight route, then repeats Step 4 and continues the current simulation; when A DDQN =[1], start to execute the penetration mode decision.

[0204] Step 6:

[0205] Calculate the output action of the lower-layer network.

[0206] Judge which maneuver form to use for penetration according to the situation of both sides at all times, and input the state space at the current moment into the intelligent agent TD3 network to obtain the output action:

[0207]

[0208] Step 7:

[0209] Penetration mission simulation. This is the step of the ballistic deviation reward evaluation function, one of the designs of the present invention.

[0210] After obtaining the action output of the penetration method in Step 6, the simulation environment conducts penetration method decision simulation according to the action. At this time, the environment conducts simulation with a fixed time step.

[0211] After obtaining the action at a certain moment, the simulation environment adjusts the action of the attacking missile. The normal acceleration of the attacking missile in the y direction executes the action The normal acceleration of the attacking missile in the z direction executes the action

[0212] After executing the current simulation time step, an immediate reward evaluation is performed. The immediate reward includes the ballistic deviation reward and the energy loss reward.

[0213] The total immediate reward is:

[0214] R j = R V + R loss

[0215] Return to the output action of Step 6 and continue the current simulation until all interceptor missiles are broken through or the attacking missile is intercepted, or it is below the lower flight bound.

[0216] Step 8:

[0217] Penetration effect evaluation.

[0218] The penetration effect evaluation of the upper-layer network is as follows:

[0219] R DDQN = R loss_all + R X + R limit_y

[0220] The penetration effect evaluation of the lower-layer network is as follows:

[0221] R TD3 = R limit_y + R x

[0222] Step 9:

[0223] Organize the "experience" obtained from this penetration mission. Organize and save the observation matrices, assigned actions, and penetration effects of the attacking missile, interceptor missile, and virtual target in this mission. Through multiple mission simulations, a simulation experience pool can be obtained, providing a basis for parameter optimization in the training of the intelligent agent network.

[0224] Repeat steps 2 to 8. After 23,000 times of training, the following training reward results are obtained, as Figure 4 , Figure 5 shown. It can be seen from the figure that as the number of simulation rounds increases, the double-layer reinforcement learning algorithm framework learns the correct penetration strategy. Around 16,000 rounds, the two networks start to converge simultaneously.

[0225] After using the trained network, the three-dimensional trajectory curves of the attacking missile and multiple interceptor missiles in the ground coordinate system are as Figure 6 and Figure 7 shown.

[0226] The control commands of the attacking missile are as Figure 8 and Figure 9 shown. It can be clearly seen from the figure that the upper-layer deep reinforcement learning outputs the penetration execution command to the attacking missile about three seconds after accurately judging the enemy's defense situation and mission timing. The lower-layer reinforcement learning then optimizes the flight path and maneuvering method of the attacking missile and quickly evades in two directions with the maximum maneuvering ability. It can be seen from Figure 10 that the attacking missile successively evades three interceptor missiles and finally successfully completes the penetration mission. Through the collaborative effect of the upper and lower layers of reinforcement learning, the system continuously optimizes the decision-making process from the feedback and realizes the best execution of the penetration mission.

[0227] The method proposed by the present invention, the single-missile penetration technology based on deep reinforcement learning, makes intelligent penetration decisions for multi-interceptor equipment interception. A double-layer deep reinforcement learning framework is designed based on the double deep Q-network algorithm and the double-delayed deep deterministic policy gradient algorithm for collaborative decision-making on penetration timing and penetration method; at the same time, the target is introduced to reduce the ballistic deviation in penetration and ensure subsequent tasks. Compared with the existing methods, this method realizes autonomous decision-making on penetration timing and breaking through the blockade of multiple interceptions on the basis of meeting successful penetration.

[0228] It should be noted that the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present invention, rather than for limiting purposes. It is easy to understand that the processes shown in the above-mentioned drawings do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0229] It should be noted that the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present invention, rather than for limiting purposes. It is easy to understand that the processes shown in the above-mentioned drawings do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0230] Further, in the embodiments of this example, an electronic device is also provided. The electronic device includes: a processor; and a memory for storing executable instructions of the processor. Wherein, the processor is configured to implement the above-mentioned three-dimensional infrared radiation field modeling method based on physical mechanism constraints when executing the executable instructions.

[0231] Figure 11 The figure shows a schematic diagram of an electronic device suitable for implementing the embodiments of the present invention.

[0232] It should be noted that Figure 11 The shown electronic device 1000 is only an example and should not impose any restrictions on the functions and usage scope of the embodiments of the present invention.

[0233] As Figure 11 As shown, the electronic device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage section 1008 into the random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operations are also stored. The CPU 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0234] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed so that a computer program read from it can be installed into the storage section 1008 as needed.

[0235] In particular, according to an embodiment of the present invention, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a storage medium, and the computer program contains program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the system of the present application are performed.

[0236] It should be noted that the storage medium shown in the embodiments of the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any storage medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the storage medium can be transmitted by any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0237] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as combinations of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0238] The units described in the embodiments of the present invention can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation to the unit itself in some cases.

[0239] It should be noted that, on the other hand, the present application also provides a storage medium, which can be included in an electronic device; or it can exist separately without being assembled into the electronic device. The above storage medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device is caused to implement the methods described in the following embodiments. For example, the electronic device can implement the steps of the method as shown in Figure 1 , Figure 2 each step of the method shown.

[0240] In one embodiment, the present application provides a computer program product, including a computer program, which when executed by a processor implements the steps in the above method embodiments.

[0241] In addition, the above accompanying drawings are only schematic illustrations of the processes included in the methods according to the exemplary embodiments of the present invention, rather than for limiting purposes. It is easy to understand that the processes shown in the above accompanying drawings do not indicate or limit the chronological order of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

Claims

1. An intelligent maneuver penetration decision-making method for an aircraft based on double-layer deep reinforcement learning, characterized in that The method includes: Constructing first self-observation state data according to the flight state parameters of the autonomous aircraft; Constructing second self-observation state data according to the flight state parameters of the interception equipment; Constructing third state space parameters according to the relative state parameters between the autonomous aircraft and the interception equipment; Constructing overall state space data based on the first self-observation state data, the second self-observation state data, and the third state space parameters; Configuring the overall state space data as the input parameters of the upper decision-making network, and using the upper decision-making network to perform reinforcement learning on the input parameters to obtain the penetration decision judgment result output by the network; wherein, the penetration decision judgment result is used to determine whether the current autonomous aircraft needs to perform maneuvering penetration.

2. The method according to claim 1, characterized in that The method further includes: Constructing fourth state space data according to the relative state parameters between the autonomous aircraft and the target, for constructing the overall state space data in combination with the fourth state space data.

3. The method according to claim 2, wherein The relative state parameters between the autonomous aircraft and the target include any one or any combination of the following: the relative distance between the autonomous aircraft and the target, the line-of-sight elevation angular velocity, the line-of-sight direction angular velocity, the line-of-sight elevation angle, and the line-of-sight direction angle.

4. The method according to claim 2 or 3, characterized in that, The method further includes: Configuring the constraint conditions corresponding to the fourth state space data; the constraint conditions include at least one of the trajectory deviation degree of the autonomous aircraft and the flight altitude of the autonomous aircraft.

5. The method according to claim 1, wherein The first self-observation state data includes the ballistic inclination angle parameter and the ballistic deflection angle parameter of the autonomous aircraft; The second self-observation state data includes the ballistic inclination angle parameter and the ballistic deflection angle parameter of the interception equipment.

6. The method according to claim 1, wherein The third state space parameters include any one or any combination of the following between the autonomous aircraft and the interception equipment: relative distance, line-of-sight elevation angular velocity, line-of-sight direction angular velocity, line-of-sight elevation angle, and line-of-sight direction angle.

7. The method according to any one of claims 1-6, characterized in that When the penetration decision judgment result output by the upper decision-making network is that the current autonomous aircraft needs to perform maneuvering penetration, the method further includes: Configuring the overall state space data as the input parameters of the lower decision-making network, and using the lower decision-making network to process the input parameters to obtain the penetration action data output by the network; wherein, the penetration action data includes the normal acceleration and / or normal overload of the autonomous aircraft.

8. The method according to claim 1, wherein The method further includes: When it is recognized that there is interception equipment targeting the autonomous aircraft, calculating the real-time distance between the autonomous aircraft and the interception equipment; When the real-time distance is less than a preset distance threshold, triggering the maneuvering penetration task of the autonomous aircraft; Executing the maneuvering penetration task of the autonomous aircraft, for constructing the overall state space data and judging whether the current autonomous aircraft needs to perform maneuvering penetration.

9. The method according to claim 1, characterized in that, The method further includes: pre-training the maneuvering penetration decision model of the autonomous aircraft, including: Initializing the basic task parameters, including: network basic parameters, guidance ratio, and penetration reward; Configuring the motion relationship between the autonomous aircraft, the interception equipment, and the target; Constructing sample overall state space data according to the sample state parameters of the autonomous aircraft and the interception equipment; Input the overall state space data of the sample into the upper-layer decision network to obtain the sample penetration decision judgment result output by the network; wherein, the upper-layer decision network is constructed based on the DDQN model; Input the overall state space data of the sample into the lower-layer decision network to obtain the sample penetration action data output by the network; wherein the lower-layer decision network is constructed based on the TD3 model; The autonomous aircraft executes the sample penetration action data to obtain the optimized trajectory data; Calculate the reduction of ballistic deviation according to the optimized trajectory data, and calculate the energy loss reward according to the penetration action data; repeat the above steps until the autonomous aircraft breaks through the interception equipment, or is intercepted by the interception equipment or is lower than the flight altitude constraint of the autonomous aircraft; Evaluate the penetration effect according to the penetration result of the autonomous aircraft to obtain the penetration evaluation result; wherein, the rewards of the upper-layer decision network include at least one of the flight lower bound limit reward, the penetration determination reward, and the energy loss reward; the rewards of the lower-layer decision network include at least one of the flight lower bound limit reward and the penetration determination reward; Construct sample data according to the overall state space data of the sample, the sample penetration decision judgment result, the sample penetration action data, and the penetration evaluation result, and save it to the experience pool; Repeat the above steps to iteratively train the autonomous aircraft maneuvering penetration decision model.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the aircraft intelligent maneuvering penetration decision method based on double-layer deep reinforcement learning according to any one of claims 1 to 9.

Images