Multi-unmanned aerial vehicle cooperative confrontation decision-making method based on dual evaluation network

Abstract

The invention discloses a multi-unmanned aerial vehicle cooperative confrontation decision-making method based on a double evaluation network, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a first local observation state and action information of a first action executed by a target unmanned aerial vehicle; inputting the first local observation state and the action information into a double-evaluation network to obtain a combined action output by the double-evaluation network; based on a reward value corresponding to the joint action, performing loop calculation on the dual-evaluation network until the reward value converges, and determining an action parameter of a second action needing to be executed by the target unmanned aerial vehicle; wherein the minimum value of the action value of the first action calculated by the first current Critic network and the action value of the first action calculated by the second current Critic network are taken in the process of double-evaluation network cycle calculation, so that parameter updating is carried out on the current Actor network. The method optimizes the game decision of the unmanned aerial vehicle.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to a multi-UAV collaborative confrontation decision-making method based on a dual evaluation network. Background Art

[0002] With the rapid development of drone technology, its application in war scenes has gradually increased. The modern war environment is becoming increasingly complex. Drones not only need to perform reconnaissance and strike missions, but also need to make real-time decisions and strategic confrontations with the enemy on a dynamic battlefield. Therefore, it is particularly important to optimize the combat game decision-making ability of drones.

[0003] Combat game theory provides an effective analytical framework for drones, which can evaluate decision-making strategies in confrontational or cooperative situations by modeling the interactive relationships between different agents.

[0004] However, in complex air combat scenarios, faced with the challenges of computational complexity and coordinated operations of multiple UAVs, existing game decisions still need to be further optimized. Summary of the invention

[0005] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a multi-UAV collaborative confrontation decision-making method based on a dual evaluation network, which optimizes the game decision of the UAVs.

[0006] In a first aspect, the present application provides a multi-UAV collaborative confrontation decision-making method based on a dual evaluation network, which is applied to a target UAV in a confrontation training between a first UAV group and a second UAV group. The method comprises:

[0007] Acquire a first local observation state and action information of the target UAV performing a first action;

[0008] Inputting the first local observation state and the action information into a dual evaluation network to obtain a joint action output by the dual evaluation network, wherein the dual evaluation network includes: a current Actor network, a target Actor network, a first current Critic network, a second current Critic network, and a target Critic network;

[0009] Based on the reward value corresponding to the joint action, the dual evaluation network is cyclically calculated until the reward value converges, and the action parameters of the second action that the target UAV needs to perform are determined;

[0010] In the process of the double evaluation network cyclic calculation, the action value of the first action calculated by the first current Critic network and the second current Critic network is minimized to update the parameters of the current Actor network.

[0011] According to an embodiment of the present application, taking the minimum value of the action value of the first action calculated for the first current critic network and the second current critic network includes:

[0012]

[0013] Where i is the number of the target drone; t is the time; k=1 represents the first current critic network, k=2 represents the second current critic network; y i,t is the minimum value of the action value of the first action of the target UAV i at time t, and the action value is used to update the parameters of the current Actor network; r i,t is the reward value of the target UAV i at time t; γ is the discount factor of the reward value; Q′ i is the target Critic network of the target UAV i; s i,t is the first local observation state of the target UAV i at time t; a i,t is the action information of the target UAV i at time t; θ i,k,Q′ is the network parameter of the target Critic network of the target UAV i; μ′ is the target Actor network of the target UAV i; θ i,k,μ′ are the network parameters of the target Actor network of the target UAV i;

[0014] The current Actor network and the target Actor network have the same network structure, and the first current Critic network, the second current Critic network, and the target Critic network have the same network structure.

[0015] According to one embodiment of the present application, after determining the action parameters of the second action that the target drone needs to perform, the method further includes:

[0016] When the target UAV performs the second action based on the action parameter, obtaining a second local observation state corresponding to the second action;

[0017] Taking a combination of the first local observation state, the action information, the reward value, and the second local observation state as a training sample, and storing the training sample in an experience replay pool;

[0018] When the number of the training samples in the experience replay pool is greater than a quantity threshold, the training samples are extracted from the experience replay pool to update the parameters of the dual evaluation network.

[0019] According to one embodiment of the present application, the action information is an action variable determined based on the maneuvering action of the target UAV in the maneuvering mode.

[0020] According to one embodiment of the present application, the maneuvering action includes constant speed level flight, accelerated level flight, decelerated level flight, fastest climb, left turn, right turn and dive.

[0021] According to one embodiment of the present application, the reward value is calculated based on the following:

[0022]

[0023] Among them, r i,t A is the reward value of the target UAV i at time t; i,t is the joint action of the target UAV i at time t; when the target UAV i belongs to the first UAV group, A i,t ∈t1 is used to indicate that the target UAV i is eliminated by the UAVs in the second UAV group at time t, A i,t ∈t2 is used to indicate that the missile launched by the target UAV i at time t has entered the state of being intercepted by the seeker, A i,t ∈t3 is used to represent that the target UAV i launches a missile when the UAV in the second UAV group enters the attackable range of the first UAV group at time t, A i,t ∈t4 is used to characterize that the rate of change of the distance between the target UAV i and the UAV j in the second UAV group at time t is a positive number.

[0024] According to one embodiment of the present application, the target critic network is used to evaluate the optimal game action.

[0025] According to an embodiment of the present application, the step of performing cyclic calculation on the dual evaluation network based on the reward value corresponding to the joint action includes:

[0026] Substituting the first local observation state and the action information into the dual evaluation network, and updating the parameters of the current Actor network and the target Actor network according to the policy gradient.

[0027] In a second aspect, the present application provides a multi-UAV collaborative confrontation decision-making device based on a dual evaluation network, which is applied to a target UAV in a confrontation training between a first UAV group and a second UAV group. The device includes:

[0028] An acquisition module, used to acquire a first local observation state and action information of the target UAV performing a first action;

[0029] A first processing module, configured to input the first local observation state and the action information into a dual evaluation network to obtain a joint action output by the dual evaluation network, wherein the dual evaluation network includes: a current Actor network, a target Actor network, a first current Critic network, a second current Critic network, and a target Critic network;

[0030] A second processing module is used to perform a cyclic calculation on the dual evaluation network based on the reward value corresponding to the joint action until the reward value converges, and determine the action parameters of the second action that the target drone needs to perform;

[0031] In the process of the double evaluation network cyclic calculation, the action value of the first action calculated by the first current Critic network and the second current Critic network is minimized to update the parameters of the current Actor network.

[0032] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network as described in the first aspect above is implemented.

[0033] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network as described in the first aspect above.

[0034] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network as described in the first aspect.

[0035] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network as described in the first aspect above.

[0036] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application.

[0037] The multi-UAV cooperative confrontation decision-making method based on a dual evaluation network provided by the present invention has the following beneficial effects compared with the prior art:

[0038] (1) By using a dual evaluation network based on an improved Actor-Critic network structure to obtain the minimum value of the action value calculated by the first current Critic network and the second current Critic network in the target UAV during adversarial training, the problem of the large action space where the action information of the training framework is located and the state space where the first local observation state is located is solved. This can effectively reduce the impact of overestimation of the action value and make a more accurate estimate of the action value of the UAV game, so that the target UAV can choose to execute the optimal second action according to the game strategy, achieve rapid and accurate decision-making in a changing battlefield environment, improve the success rate of combat mission execution, promote the application of UAVs on future battlefields, and provide more efficient support for war operations.

[0039] (2) In response to the joint actions of target UAVs, a refined reward mechanism for UAV confrontation games was constructed to guide UAVs to learn and form flexible and effective aerial game strategies, achieve rapid and accurate decision-making in a changing battlefield environment, improve the success rate of combat mission execution, and build a more advanced multi-UAV confrontation mission training system.

[0040] (3) By delaying the Actor network update strategy to minimize the error before the Actor network parameters are updated, the update frequency of the Actor network is set lower than the update frequency of the Critic network, reducing error accumulation and the possibility of using the value valuation obtained from the unchanged Critic network to guide repeated updates of the Actor network, thereby achieving higher quality policy updates and reducing the impact of minimizing operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0042] Figure 1 It is a structural diagram of the adversarial scenario model provided in an embodiment of the present application;

[0043] Figure 2 This is one of the flow charts of the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network provided in an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of the structure of a dual evaluation network provided in an embodiment of the present application;

[0045] Figure 4 This is the second flow chart of the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network provided in an embodiment of the present application;

[0046] Figure 5It is a structural schematic diagram of a multi-UAV collaborative confrontation decision-making device based on a dual evaluation network provided in an embodiment of the present application;

[0047] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0049] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0050] In combination with the accompanying drawings, the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network, the multi-UAV collaborative confrontation decision-making device based on a dual evaluation network, the electronic device and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0051] Among them, the multi-UAV collaborative confrontation decision-making method based on the dual evaluation network can be applied to the terminal, and can be specifically executed by the hardware or software in the terminal.

[0052] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0053] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0054] The embodiment of the present application provides a multi-UAV collaborative confrontation decision-making method based on a dual evaluation network. The executor of the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network can be an electronic device or a functional module or functional entity in the electronic device that can implement the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The multi-UAV collaborative confrontation decision-making method based on a dual evaluation network provided in the embodiment of the present application is explained below using an electronic device as an example of the executor.

[0055] The method is applied to any UAV in the confrontation training between the first UAV group and the second UAV group, and the UAV is taken as the target UAV.

[0056] The method provided by the present invention does not require an environmental model of the confrontation environment and a communication model for multi-UAV collaboration, but completes training through interaction with a simulation environment. Taking the construction of a simulation environment for confrontation training between a first UAV group and a second UAV group in a three-dimensional situation combat system as an example, the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network of the present application is explained below, which is not regarded as limiting the scope of protection of the present application.

[0057] It should be noted that the first drone group can be marked as the red side (our side), including 4 drones, and the second drone group can be marked as the blue side (enemy side), including 4 drones.

[0058] The simulation environment of the adversarial training includes a first drone swarm and a second drone swarm, as well as an adversarial scenario model of the adversarial training.

[0059] In confrontation training, the blue team's radar detection range can be set to 80km, and the red team's radar detection range can be set to 60km. When one party's drone enters the other party's radar detection range, it will be successfully detected by the other party's radar, and the two sides will coordinate in attack and defense.

[0060] like Figure 1 As shown, the confrontation scenario model is set in a closed three-dimensional rectangular space ranging from 118 to 120 degrees east longitude and 23 to 25 degrees north latitude with an upper altitude of 15 kilometers, forming a simulated combat space with boundary restrictions, without considering terrain occlusion, and setting corresponding confrontation parameters. The air combat area has a minimum flight altitude restriction (flight altitude limit), and the minimum damage altitude is 2 kilometers.

[0061] For example, the confrontation parameters may include the following information: the first drone group (red side) carries detection sensors (radars) and 6 missiles, the second drone group (blue side) carries detection sensors and 8 missiles, both sides support AIM120C missile types, the radar is set to have only forward detection function, the RCS coefficient is 0.03, and the forward detection azimuth and pitch angles are both ±60 degrees. The effective detection range of the forward radar of the first drone group (red side) is 60 kilometers, and the effective detection range of the forward radar of the second drone group (blue side) is 80 kilometers.

[0062] The multi-UAV confrontation process is instantiated as an air-to-air battle, and the enemy and our UAVs are placed in a three-dimensional space (confrontation scenario model) to build a simulation environment for confrontation training.

[0063] Among them, based on the conversion of the battlefield longitude and latitude settings in the rules, the space size is approximately 202.211km×333km×15km. The red and blue sides each contain 4 drones, and the drones of both sides are deployed according to the test simulation cases.

[0064] For example, when the test simulation case is test case 1, the arrangement method is that the red and blue drones are arranged in a "one" shape on the same longitude. When the test simulation case is test case 2, the arrangement method is that the red drone is still arranged in a "one" shape, and the blue drones are grouped in pairs, and each group is distributed up and down.

[0065] After the confrontation training begins, the drones of both sides fly in the specified heading direction. Each drone executes the multi-UAV collaborative confrontation decision-making method based on the dual evaluation network, changes its own motion mode based on its own algorithm recognition and confrontation strategy, and the two sides conduct aerial combat collaborative game confrontation based on actual combat simulation.

[0066] When the red and blue sides meet in the air, the drone's radar identifies and locks onto the other drone, and launches missiles, resulting in a fierce confrontation and attack and defense game. The simulation environment will make real-time judgments. If the attack hits the other drone, the drone will be given a certain reward; conversely, if it is hit by the other drone, the hit drone will be removed from the simulation environment and given a certain penalty.

[0067] The attack range of the drone is determined by the attack radius r and the attack angle ω. The fan-shaped area is composed of an attack radius r set to 20km, and the attack range is formed by the attack radius r, and the attack angle ω, Set to

[0068] It should be noted that 60km and 80km are the radar detection distances of the red drone and the blue drone respectively. The radius range with radar detection distance can be used as the radar field of view of the drone. When one or more enemy drones enter the radar field of view, our drone can identify one or more enemy drones. Our single drone can only lock one enemy drone and have the conditions to attack after locking the enemy drone. The 20km corresponding to the attack radius r is the range distance with a high probability of hitting the mounted missile. Therefore, when the locked enemy aircraft enters the attack range of 20km, the drone that locks the enemy aircraft will launch a missile at it, but it may not hit it.

[0069] For the drones of the first drone group (the red side), when the distance to the drones of the second drone group (the blue side) is less than the radar field of view r1 (r1 = 60km) and the angles θ1, At the same time, if the drones of the second drone group (blue side) meet the conditions of appearing and being locked in the radar field of view r1, it will continue to judge whether the locked drones of the second drone group (blue side) have entered the 20km attack range of the first drone group (red side). If the locked drones of the second drone group (blue side) enter the attack range, the missile will be launched;

[0070] For the drones of the second drone group (blue side), when the distance to the drones of the first drone group (red side) is less than the radar field of view r2 (r2 = 80km) and the angle θ2, At the same time, if the drones of the first drone group (red side) meet the conditions of appearing and being locked within the radar field of view r2, it will continue to judge whether the locked drones of the first drone group (red side) have entered the 20km attack range of the second drone group (blue side). If the locked drones of the first drone group (red side) enter the attack range, the missile will be launched.

[0071] At the same time, taking into account that each drone has been stealthed, the radar cross section (RCS) coefficient of the drone is included in the confrontation parameters of the confrontation training. Among them, the forward RCS coefficient of both the enemy and our side is set to 0.03 in the simulation platform. The result after the coefficient setting has been reflected in the radar scanning result and is directly exported by the simulation platform through the interface.

[0072] During the confrontation training, the drone of either party will attack the enemy targets that have been detected by the radar and the enemy drones that have entered its attack range. The probability of hitting the enemy drone depends on the actual simulation, the training process and the test results, and is roughly within [0.4, 0.8].

[0073] Assuming that the red and blue sides are on the same horizontal plane, the confrontation scene is set in three-dimensional space to construct a simulation environment for confrontation training. Only the speed, acceleration and heading of each drone are considered. The motion equation of the drone can be written as follows:

[0074] v t+1 =v t +a t t

[0075] α t+1 =α t +λ t

[0076] β t+1 =β t +μ t

[0077] x t+1 =x t +v t cos(α t )sin(β t )t

[0078] y t+1 =y t +v t sin(α t )sin(β t )t

[0079] z t+1 =z t +v t cos(α t )t

[0080] Among them, v t represents the flight speed at time t; v t+1 represents the flight speed at time t+1; a t represents the flight acceleration at time t; α t represents the yaw heading at time t; α t+1 Indicates the yaw heading at time t+1; β t represents the pitch heading at time t; β t+1 represents the pitch heading at time t+1; t represents the yaw steering angle at time t; μ t represents the pitch steering angle at time t; x t ,y t 、z t represents the coordinates at time t; x t+1 ,y t+1 、z t+1 Represents the coordinates at time t+1.

[0081] like Figure 2 As shown, the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network includes: steps 210 to 230.

[0082] Step 210: Obtain a first local observation state and action information of the target UAV performing a first action.

[0083] Among them, any drone in the first drone group (red side) or the second drone group (blue side) is used as the target drone i.

[0084] The first action can be the action performed by the target drone i at time t.

[0085] It can be understood that the battlefield situation information observable by the target UAV i mainly includes position and heading information. This information corresponding to the first action can be used as the first local observation state of the target UAV i. For the target UAV i, its observation information can be written in the 16-dimensional state space in the form of (machine number, longitude, latitude, altitude, yaw angle, pitch angle, roll angle, north speed, east speed, north acceleration, celestial acceleration, east acceleration, Mach number, normal overload, number of existing missiles, and maximum launch distance).

[0086] Taking target drone i as an example, corresponding to the 16-dimensional state space, the observation information o of target drone i i Set to a 16-dimensional tuple as follows:

[0087]

[0088] Among them, i Indicates the observation information of target UAV i; id i Indicates the number of the target drone i; lo i ,la i ,al i represents the longitude, latitude and altitude of the target drone i; y i ,p i ,r i represents the yaw angle, pitch angle and roll angle of the target UAV i; v ei represents the northward speed and eastward speed of the target UAV i; a Ui ,a Ei represents the north acceleration of target UAV i; the sky acceleration and the east acceleration; Ma i represents the Mach number of the target UAV i; N i represents the normal overload of target UAV i; M i ,A iThey represent the existing number of missiles and the maximum launch distance of target UAV i respectively.

[0089] For each UAV, each type of maneuver information is implemented by 18 variables as control parameters, which can be expressed in the form of (bApplyNow, iCmdID, speed type, Mach number, altitude, overload, track inclination, slope, heading, body axis heading, maneuver time, throttle, threshold value, tilt plane angle, body axis pitch, body axis, fCmdBdyPsiDeg, steering).

[0090] It should be noted that o i is the observation space, which represents the information obtained by the target UAV i, s t It is the state space, which represents the global information obtained by all drones and is a complete description of the entire system at time i.

[0091] The observation spaces of all drones can be spliced ​​into a state space s t ,Right now:

[0092] {o1,o2,…,o N}=s t

[0093] Where N is the number of drones in the fleet where the target drone is located, o i and i,t It represents a content, which is the observation information obtained by the target UAV i.

[0094] o i Mainly used to store in the experience pool, after which the Actor network will be based on the observation information of each drone. i To select an action, i,t It is the first local observation state, and the global state space is mainly used to update the Critic network, that is, to update the strategy.

[0095] Taking the target drone i as an example, its action information a i It can be set as an 18-dimensional tuple as follows:

[0096] a i =(bApplyNow i ,iCmdID i ,iVelType i ,fCmdMa i ,fCmdAlt i ,fCmdNy i ,fCmdPitchDeg i ,fCmdPhi i ,fCmdHeadingDeg i,fCmdDeltaHeading i ,fTime i ,fCmdThrust i ,fThrustLimit i ,fCmdObliqueDeg i ,fCmdBdyPithDeg i ,fCmdBdyPsiDeg i ,fCmdRollDeg i ,iTurnDirection i )

[0097] bApplyNow i 、iCmdID i and fCmdRollDeg i is the identity information of the target drone i, bApplyNow i Control jump parameters for the maneuver number of target drone i, identify the rising edge from False to True to send the remaining control parameters to target drone i, iCmdID i fCmdRollDeg is the maneuver sequence number of target UAV i, i is the target attitude roll angle of target drone i; iVelType i fCmdMa is the control speed type of target drone i; i is the control Mach number of target UAV i; fCmdAlt i fCmdNy is the control height of target UAV i; i fCmdPitchDeg is the control overload of the target drone i; i fCmdPhi is the control track inclination angle of target UAV i; i fCmdHeadingDeg is the control slope of target UAV i; i fCmdDeltaHeading is the control heading of the target UAV i; i is the control body axis heading of target UAV i; fTime i fCmdThrust is the control maneuver time of target UAV i; i fThrustLimit is the throttle control for the target drone i; i fCmdObliqueDeg is the control threshold of target UAV i; i fCmdBdyPitchDeg is the control tilt angle of the target drone i; i fCmdBdyPsiDeg is the control body axis pitch of target UAV i;i is the control body axis of the target drone i; iTurnDirection i Control and steer the target UAV i.

[0098] In this step, the first local observation state s corresponding to the first action performed by the target UAV i at time t is collected i,t and action information a i,t .

[0099] Step 220: Input the first local observation state and the action information into a dual evaluation network to obtain a joint action output by the dual evaluation network, wherein the dual evaluation network includes: a current Actor network, a target Actor network, a first current Critic network, a second current Critic network, and a target Critic network.

[0100] like Figure 3 As shown, the dual evaluation network is an Actor-Critic network structure, and the combination of the first current Critic network and the second current Critic network can be used as a double-layer current Critic network.

[0101] In actual execution, the first local observation state s i,t and action information a i,t Input to the current Actor network in the dual evaluation network, the first local observation state s is evaluated by the current Actor network and the target Actor network i,t and action information a i,t Calculate and select the distribution probability of the second action as the action strategy to obtain the joint action A corresponding to the first action performed by the target UAV i at time t i,t And output.

[0102] Among them, the joint action A corresponding to the first action of target UAV i i,t It is used to characterize the game state of the target UAV i in the adversarial training, which may include the task completion status of the target UAV i, for example, whether it hits the opponent UAV or is hit by the opponent.

[0103] Step 230: Based on the reward value corresponding to the joint action, the dual evaluation network is cyclically calculated until the reward value converges, and the action parameters of the second action that the target drone needs to perform are determined.

[0104] In the process of the double evaluation network cyclic calculation, the action value of the first action calculated by the first current Critic network and the second current Critic network is minimized to update the parameters of the current Actor network.

[0105] Among them, the simulation environment takes the joint action A of the target UAV i at time t as i,t The corresponding game state uses a custom reward function to obtain the reward value r of the target drone i at time t i,t .

[0106] The reward function is used to characterize the mapping relationship between joint actions and reward values.

[0107] It can be understood that the action parameters can be used to configure parameters of the target drone so that the target drone can perform the second action.

[0108] In this embodiment, the current Actor network is used to generate actions, and outputs a selected action a corresponding to a probability distribution of an action according to the current state s and the first action; during the update of the current Actor network, the strategy is adjusted to improve future returns according to the reward value and feedback of the new state.

[0109] The target Actor network is used to provide a stable target strategy, and outputs the action a′ selected under the target strategy according to the target state s′ (the next state used during training); during the update of the target Actor network, the parameter update remains synchronized with the current Actor network, making the learning process more stable.

[0110] The current Critic network is used to evaluate the value of the current strategy. According to the current state s and the first action a, it outputs the state-action value function Q(s,a) or the state value function V(s). During the update of the current Critic network, the network parameters are updated by calculating the TD (Temporal Difference) error to more accurately estimate the value of the action.

[0111] The target critic network is used to provide a stable value estimate. It generates the target value Q'(s',a') or V'(s') according to the target state s' and the target action a' generated by the target actor network.

[0112] During the target Critic network update process, it helps calculate a more stable target value, reduces the variance in the learning process, and keeps consistent with the network parameters of the current Critic network through soft updates.

[0113] It should be noted that taking the minimum value of the action value of the first action calculated by the first current critic network and the second current critic network can reduce the impact of overestimation of the action value.

[0114] According to the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network provided in an embodiment of the present application, in the target UAV of the confrontation training, the dual evaluation network obtained based on the improved Actor-Critic network structure is used to take the minimum value of the action value calculated by the first current Critic network and the second current Critic network, thereby solving the problem that the action space where the action information of the training framework is located and the state space where the first local observation state is located are large, and can effectively reduce the impact of the overestimation of the action value, and make a more accurate estimate of the action value of the UAV game, so that the target UAV chooses to execute the optimal second action according to the game strategy, thereby achieving fast and accurate decision-making in a changing battlefield environment, improving the success rate of combat mission execution, promoting the application of UAVs on future battlefields, and providing more efficient support for war operations.

[0115] In some embodiments, the motion information is a motion variable determined based on a maneuvering motion of the target drone in a maneuvering mode.

[0116] In actual execution, the action information a i The value ranges of the 18 variables in are constantly changing according to the actual flight conditions of the target UAV i.

[0117] Having an action space with 18 variables will greatly reduce the exploration efficiency, resulting in the drone being unable to fully explore all possible actions within a limited time and training process, and will also lead to a decrease in strategy accuracy, making it impossible to learn a stable optimal strategy.

[0118] Therefore, the action variables of the maneuvering mode can be used to replace the control parameters corresponding to the 18 variables of the target UAV to achieve dimensionality reduction of the action space.

[0119] For example, the maneuvering mode can cover the flight actions and strategies of the UAV in confrontation training. The maneuvering mode can include cruise mode, climb / descent mode, turning mode, emergency stop mode, dodge mode and tracking mode, etc.

[0120] In this embodiment, the action variables of the maneuvering mode are selected to replace the control parameters of the action space, which can reduce the complexity of the action space, shorten the training learning convergence time, reduce the complexity of the dual evaluation network, and also make the dual evaluation network more interpretable and applicable.

[0121] In some embodiments, the maneuvers include constant speed level flight, accelerated level flight, decelerated level flight, fastest climb, left turn, right turn and dive.

[0122] Among them, the maneuvering mode can be adjusted according to mission requirements, flight environment and operational requirements.

[0123] In actual implementation, in order to adapt to the scenario of UAV confrontation training, the maneuvering mode can include seven maneuvers: constant speed level flight, accelerated level flight, decelerated level flight, fastest climb, left turn, right turn, and dive. The maneuvering variables corresponding to each maneuvering action have different control parameters, thereby realizing the control of the UAV's maneuvering actions.

[0124] After reducing the 18-dimensional action space to 7 dimensions, the control parameters corresponding to the action variables of each maneuver in constant speed level flight A1, accelerated level flight A2, decelerated level flight A3, fastest climb A4, right turn A5, left turn A6 and dive A7 are shown in Table 1:

[0125] Table 1 Correspondence between action variables and control parameters

[0126]

[0127]

[0128] In this embodiment, the flight of the target UAV is abstracted through maneuvering actions, which can improve control efficiency and facilitate mission planning and coordination of the target UAV.

[0129] In some embodiments, the reward value is calculated based on the following:

[0130]

[0131] Among them, r i,t A is the reward value of the target UAV i at time t; i,t is the joint action of the target UAV i at time t; when the target UAV i belongs to the first UAV group, A i,t ∈t1 is used to indicate that the target UAV i is eliminated by the UAVs in the second UAV group at time t, A i,t ∈t2 is used to indicate that the missile launched by the target UAV i at time t has entered the state of being intercepted by the seeker, A i,t ∈t3 is used to represent that the target UAV i launches a missile when the UAV in the second UAV group enters the attackable range of the first UAV group at time t, A i,t ∈t4 is used to characterize that the rate of change of the distance between the target UAV i and the UAV j in the second UAV group at time t is a positive number.

[0132] It is understandable that the first drone group is the red team. i,t ∈t1, the target drone i is destroyed by any blue drone at time t, then the target drone i will receive a penalty and a reward value r i =-300; if A i,t∈t2, the missile launched by the target UAV i at time t enters the state of interception by the seeker, indicating that the missile launched by the target UAV i at time t has successfully locked the target, then the target UAV i gets a reward, and the reward value is r i =+100; if A i,t ∈t3, the target drone i launches a missile when the blue drone enters the attack range at time t, then the target drone i gets a reward, the reward value is r i =+100; if A i,t ∈t4, the distance change rate between target drone i and blue drone j at time t is positive, then target drone i gets a reward, and the reward value is r i =+1.

[0133] In this embodiment, a refined reward mechanism for UAV confrontation game is constructed for the joint actions of target UAVs, which guides UAVs to learn and form flexible and effective air game strategies, achieves fast and accurate decision-making in a changing battlefield environment, improves the success rate of combat mission execution, and can build a more advanced multi-UAV confrontation mission training system.

[0134] In some embodiments, Figure 3 As shown, taking the minimum value of the action value of the first action calculated by the first current critic network and the second current critic network includes:

[0135]

[0136] Where i is the number of the target drone; t is the time; k=1 represents the first current critic network, k=2 represents the second current critic network; y i,t is the minimum value of the action value of the first action of the target UAV i at time t, and the action value is used to update the parameters of the current Actor network; r i,t is the reward value of the target UAV i at time t; γ is the discount factor of the reward value; Q′ i is the target Critic network of the target UAV i; s i,t is the first local observation state of the target UAV i at time t; a i,t is the action information of the target UAV i at time t; θx ,k,Q′ is the network parameter of the target Critic network of the target UAV i; μ′ is the target Actor network of the target UAV i; θ i,k,μ′ are the network parameters of the target Actor network of the target UAV i;

[0137] The current Actor network and the target Actor network have the same network structure, and the first current Critic network, the second current Critic network, and the target Critic network have the same network structure.

[0138] On the basis of the Actor-Critic network structure, a Critic network with the same structural parameters as the original Critic network is added to obtain a double-layer Critic network consisting of a first current Critic network and a second current Critic network. A minimization operation is performed between the double-layer Critic networks to reduce the impact of overestimation of action value.

[0139] For target drone i, the objective function of the first Critic network for parameter update is as follows:

[0140]

[0141] Among them, y i,t,1 and i,t,2 are the value evaluations of the first action performed by the target UAV i at time t by the first current critic network and the second current critic network respectively; r i,t is the reward value of the target UAV i at time t; γ is the discount factor of the reward value; Q i′ is the target critic network of target UAV i; s i,t is the first local observation state of target UAV i at time t; a i,t is the action information of the target UAV i performing the first action at time t; θ i,1,Q′ is the network parameter of the first current critic network of target UAV i; θ i,2,Q′ is the network parameter of the second current critic network of target UAV i; μ′(s i,t |θ i,μ′ ) and Q′ i (s i,t ,a i,t |θ i,Q′ ) are the target Actor network and target Critic network with the same structure as the current Actor network and the current Critic network respectively; θ i,μ′ and θ i,Q′ They are the network parameters of the target Actor network and the target Critic network respectively.

[0142] Based on the above objective function, we take the smaller value of the action value of the first action calculated by the first current critic network and the second current critic network to obtain y i,t, and set it as the objective function value for parameter updates of the current Actor network and the target Critic network.

[0143] In this embodiment, by taking the minimum value of the action value of the first action calculated by the first current critic network and the second current critic network, the estimation error caused by the overestimation of the action value of the drone can be effectively reduced, thereby improving the efficiency of the drone adversarial training.

[0144] In some embodiments, after determining the action parameters of the second action that the target drone needs to perform, the method further includes:

[0145] When the target UAV performs the second action based on the action parameter, obtaining a second local observation state corresponding to the second action;

[0146] Taking a combination of the first local observation state, the action information, the reward value, and the second local observation state as a training sample, and storing the training sample in an experience replay pool;

[0147] When the number of the training samples in the experience replay pool is greater than a quantity threshold, the training samples are extracted from the experience replay pool to update the parameters of the dual evaluation network.

[0148] In order to reduce the accumulation of model training errors, the embodiments of the present application adopt a delayed Actor network update strategy to optimize the entire adversarial training learning process and minimize the error before updating by reducing the update frequency of the Actor network.

[0149] The error refers to the error between the output action of the Actor network and the optimization target, that is, the action deviation caused by not fully learning the feedback of the Critic network. The delayed Actor network update strategy will cause the parameters of the Actor network to be synchronized at a certain frequency instead of being updated directly every time, which can reduce the estimation error of the Critic network feedback.

[0150] In actual execution, after configuring the action parameters of the second action for the target UAV, the target UAV executes the second action at time t+1 and collects the second local observation state s corresponding to the second action. i,t+1 .

[0151] The first local observation state s i,t , action information a i,t , reward value r i,t and the second local observation state s i,t+1 .

[0152] In actual execution, an internal variable G(s i,t , a i,t ) is defined to record the state s i,t - action a i,t pair of a single Critic network. The number of pairs of the state s i,t - action a i,t is used as the number of times the double evaluation network evaluates the value of the pair (s i,t , a i,t ).

[0153] Assume that under n consecutive time steps, the double-layer Critic network of each agent (drone) evaluates the value of the pair (s i,t , a i,t ) m times. Then, there are local observation states at the (l1, l2, l3,..., l m )-th time steps respectively and rewards

[0154] For the time step at the current t moment, the update of the action parameters and the evaluation of the action value of the pair (s i,t , a i,t ) corresponding to the first action are delayed, making m < n to ensure a certain continuity of the action. The specific update strategy formula is as follows

[0155]

[0156] Among them, G(s i,t , a i,t ) mod n = 0 means that the internal variable G(s i,t , a i,t ) can be divided evenly by n.

[0157] In this embodiment, through the delayed Actor network update strategy, the error before the update of the Actor network parameters is minimized. The update frequency of the Actor network is set to be lower than that of the Critic network, reducing error accumulation and the possibility of guiding the repeated update of the Actor network using the value estimation obtained from the unchanged Critic network, thereby achieving higher-quality policy updates and reducing the impact of the min operation.

[0158] In some embodiments, the target Critic network is used to evaluate the optimal game action.

[0159] In this embodiment, in order to separate the selection of the optimal game action and the value estimation, the current Critic network is used to select the optimal game action, and the target Critic network is used to evaluate the optimal game action.

[0160] In actual execution, set the simulation step size t s is 0.04s, and the decision step length is t p 4s, the total number of steps in a single round of training for Deformable Convolutional Networks-Multi-Agent DeepDeterministic Policy Gradient (DCN-MADDPG) is t max is 100, the total number of training rounds is N max is 400, the experience storage area size is R b is 100000.

[0161] The embodiment of the present application adopts the DCN network to solve the action generation and evaluation of the continuous action space.

[0162] In the dual evaluation network, the combination of the current Actor network and the target Actor network is taken as the Actor network. In the Actor network, the number of neurons in the input layer is determined by the dimension of the state space. The dimension of the state space is 16, and correspondingly, the number of neurons is 16. The output layer is determined by the dimension of the action space. The dimension of the action space is 7 dimensions, and correspondingly, the number of neurons in the output layer is 7.

[0163] Among them, both the current Actor network and the target Actor network are set with two hidden layers, the number of neurons in each hidden layer is 64, and the activation function is selected as ReLU.

[0164] The evaluation layer of the DCN network is the Critic network, including the first current Critic network, the second current Critic network and the target Critic network with the same parameter settings. The number of neurons in its input layer is the sum of the input layer and the output layer of the Actor network, which is 23. The number of neurons in the output layer is 1. Two hidden layers are also set, and the number of neurons in each hidden layer is 64. The activation function also selects ReLU. The final result is obtained by taking the smaller value of the action value calculated by the objective function of the first current Critic network and the second Critic network. The game confrontation simulation parameter settings are shown in Figure 2:

[0165] Table 2 Game confrontation simulation parameter settings

[0166] parameter symbol Parameter Value Simulation step size / s <![CDATA[t s ]]> 0.04 Decision step length / s <![CDATA[t p ]]> 4 Maximum step length for single round training <![CDATA[t max ]]> 100 Total number of training rounds <![CDATA[N max ]]> 400 Experience storage area size <![CDATA[R b ]]> 100000

[0167] like Figure 4As shown in the figure, set the number of training rounds, the capacity of the experience replay pool, the number of agents (drones), the learning rate of the Actor network, the learning rate of the Critic network, the delay update frequency, etc. And set the Actor network and Critic network parameters in each agent (drone), the current training round b = 1, and the time step r = 1.

[0168] Among them, the parameter settings of the dual evaluation network are shown in Table 3. The learning rate of the Actor network and the Critic network is set to 0.01, the reward discount factor is set to 0.95, and the soft update parameter is set to 0.01.

[0169] Table 3. Dual evaluation network parameter settings

[0170] Parameter name Parameter Value Learning rate of the Actor network 0.01 Learning rate of the critic network 0.01 Reward Discount Factor 0.95 Soft Update Parameters 0.01

[0171] During the confrontation training between the first drone swarm and the second drone swarm, the drone obtains the local observation state in the simulation environment through the state interface of the three-dimensional situation system, and combines its own algorithm strategy framework to interact its own action information and local observation state with the simulation environment, thereby completing the entire algorithm process.

[0172] Initialize the initial state s1 of the simulation environment, and the Actor network of the dual evaluation network in any target UAV i is initialized according to the first local observation state s i,t ; Give the action information a of the first action i,t , and combined into joint action A i,t ;

[0173] The simulation environment is based on the joint action A i,t Calculate the reward r of target drone i at time t i,t , execute the second action, and enter the second local observation state s at the next time t+1 i,t+1 ;

[0174] At the next time t=t+1, the experience sample is stored as the experience sample (s i,t , A i,t , r i,t ,s i,t+1 ) to the experience replay pool;

[0175] When t>T and the number of training rounds b>B, the adversarial training ends.

[0176] When t>T and the number of training rounds b≤B, enter the next round of adversarial training;

[0177] When t≤T and the number of experience samples in the experience replay pool reaches D, M samples are randomly selected from the experience replay pool to update the network parameters of the Critic network of each agent (drone);

[0178] In some embodiments, the step of performing a cyclic calculation on the dual evaluation network based on the reward value corresponding to the joint action includes:

[0179] Substituting the first local observation state and the action information into the dual evaluation network, and updating the parameters of the current Actor network and the target Actor network according to the policy gradient.

[0180] In actual execution, the Critic network aims to minimize the value assessment error and updates the network parameters θ of the Critic network. i,Q , each drone calculates the network parameters θ of its own Actor network i,μ The policy gradient And update the network parameters θ of the Actor network by gradient descent i,μ , the calculation formula is as follows:

[0181]

[0182] Among them, Δθ i,Q is the network parameter θ of the Critic network i,Q Correction value; L(θ i,Q ) is the loss function; Θ D is the expected value of the batch sample; Critic Network Q i (s i,t ,a i,t |θ i,Q ) for action information a i,t Find partial derivatives; For the Actor network μ i (s i,t |θ i,μ ) for network parameters θ i,μ Find the partial derivative.

[0183] Calculate the policy gradient of the network parameters of the Actor network of each agent (drone) and update the network parameters of the Actor network;

[0184] The network parameters of the target Critic network and the target Actor network are soft updated, and the updated dual evaluation network enters the next round of adversarial training.

[0185] The weight of the original network parameters is controlled by the update coefficient τ, and the target Critic network and the target Actor network are soft updated, as shown in the following formula:

[0186]

[0187] Among them, θi,k,Q′ is the network parameter of the target critic network of target UAV i; τ is the update coefficient; when k = 1, θ i,k,Q is the network parameter of the first current critic network of target UAV i. When k = 2, θ i,k,Q is the network parameter of the second current critic network of target UAV i; θ i,μ′ is the network parameter of the target Actor network of target UAV i; θ i,μ is the network parameter of the current Actor network of target drone i.

[0188] For the red team drone, the DCN-MADDPG algorithm is used for training, the Actor network is used as the drone's decision model, and the Critic network is used as the drone's evaluator.

[0189] The DCN-MADDPG algorithm adopts a centralized training and decentralized execution model.

[0190] During the training of the dual evaluation network, the first local observation state s of all target drones i on the red team is input to the Critic network. i,t and action information a i,t , outputs a Q value, which is used to evaluate the decision made by the Actor network, and according to the reward value r fed back by the simulation environment i,t Update the network parameters of the Critic network.

[0191] Then, based on the Q value provided by the double-layer current Critic network, the policy gradient is calculated to update the Actor network.

[0192] During execution, the Actor network directly obtains the first local observation state s of the battlefield situation from the environment. i,t Do and action information a i,t The dual evaluation network outputs the joint action A i,t , execute the corresponding second action.

[0193] For example, the adversarial training scenario includes two drone clusters, red and blue. Each side has 4 drones. The red drone uses the MB-MADDPG algorithm, and the blue drone uses a method based on unknown rules.

[0194] The initial heading of the Red UAV is set to 180°, and the initial speed is set to 0.9Ma. The initial heading of the Blue UAV is set to 0°, and the initial speed is set to 0.9Ma. The two sides fly towards each other. During the confrontation, the observation radius of the Red UAV is set to 60km, and the attack angle is set to In contrast, the observation radius of the blue drone is set to 80km and the attack angle is set to

[0195] Here, two test cases with different initial positions and different blue team strategies are used for simulation testing.

[0196] In test case 1, the drone combat scenario is limited to the area between 118 and 120 degrees east longitude and 23 and 25 degrees north latitude, with an upper altitude limit of 15 kilometers. The air combat area has a minimum flight altitude limit and the minimum damage altitude is 2 kilometers.

[0197] In test case 1, the initial parameter configurations of the four drones of the red team are shown in Table 3, and the initial parameter configurations of the four drones of the blue team are shown in Table 4. The initial altitude of the drone is m_alt = 10000, the initial speed is m_speed_m_s = 240, the initial oil volume is m_oil = 5000, and the maximum simulation time is 60 minutes. In addition, the forward RCS coefficients of both the enemy and our side are 0.03. The value ranges of the headings α and β are [-π, π], and the value ranges of the steering angles λ and μ are [-π, π], that is, the maximum angle of each steering of the drone is π.

[0198] Table 3 Red team initial parameter configuration

[0199]

[0200]

[0201] Table 4 Initial parameter configuration of the blue team

[0202] Blue Square Latitude and longitude / (longitude, latitude) Flight heading / ° Number of bombs Detection range / m Plane 5 (119.2，23.2) 0 8 80000 Plane 6 (119.4，23.2) 0 8 80000 Plane 7 (119.6，23.2) 0 8 80000 Plane 8 (119.8，23.2) 0 8 80000

[0203] On the basis of successfully attacking all four drones of the blue side, our side survived one drone, the exchange ratio was 1.25, the effective attack ratio was 1.0, the effective attack rate was 0.9, the attack success rate was 1.0, and the final score was 205.10. The final training results of our side facing the blue side strategy 1 are shown in Table 5:

[0204] Table 5. Training game confrontation results based on Blue Team Strategy 1

[0205]

[0206] In test case 2, the initial position of the blue drone adopts the initial arrangement of "two by two" groups, and the two drones in each group are arranged in a vertical queue. The attack strategy adopts a rule-based constraint algorithm. When the drone radar does not scan the enemy aircraft, the drone performs forward cruising, launches missiles when the enemy aircraft enters the attack range, and when the missile enters the autonomous guidance stage, the drone executes the return command and turns around to return to the initial position.

[0207] In test case 2, the initial parameter configurations for the 4 drones of the red team are shown in Table 6, and the initial parameter configurations for the 4 drones of the blue team are shown in Table 7:

[0208] Table 6 Red team initial parameter configuration

[0209] Red Party Latitude and longitude / (longitude, latitude) Flight heading / ° Number of bombs Detection range / m Initial height / m Plane 1 (118.2，25.8) 180 6 60000 10000 Plane 2 (118.4，25.8) 180 6 60000 10000 Plane 3 (118.6，25.8) 180 6 60000 10000 Plane 4 (118.8，25.8) 180 6 60000 10000

[0210] Table 7 Initial parameter configuration of the blue team

[0211] Blue Square Latitude and longitude / (longitude, latitude) Flight heading / ° Number of bombs Detection range / m Initial height / m Plane 5 (118.3，23.2) 0 8 80000 10000 Plane 6 (118.3，23.2) 0 8 80000 12000 Plane 7 (118.7，23.2) 0 8 80000 10000 Plane 8 (118.7，23.2) 0 8 80000 12000

[0212] Similar to strategy 1, our strategy can obtain a good score. When the blue team's strategy is more flexible, our team survives 2 drones, with an exchange ratio of 1.33, an effective attack ratio of 0.75, an effective attack efficiency of 0.58, an attack success rate of 1.0, and a final score of 195.66. The training game confrontation results based on the blue team's strategy 2 are shown in Table 8:

[0213] Table 8 Training game confrontation results based on Blue Team Strategy 2

[0214]

[0215] A reinforcement learning training framework based on a three-dimensional combat situation visualization monitoring system is constructed. The diverse maneuvering modes of UAVs are integrated, and the continuous state changes of UAVs and their missiles are read and responded to in real time. A high-dimensional observation space for UAV collaborative combat is constructed, which improves the applicability of reinforcement learning algorithms in high-latitude environments and provides a new strategy learning approach for multi-UAV collaborative combat.

[0216] In this embodiment, in order to solve the technical problem of multi-UAV collaborative confrontation decision-making in the context of complex air combat, a DCN-MADDPG method based on the improvement of the three-dimensional situation system is proposed from the perspective of multi-agent reinforcement learning method. This method is based on the simulation platform of the three-dimensional combat situation visualization monitoring system, and constructs a high-dimensional model training framework, which comprehensively considers the diverse maneuvering modes of UAVs, can read and respond to the continuous state changes of UAVs and their missiles in real time, and construct a high-dimensional observation space. For complex action instructions, the MADDPG algorithm with a double-layer Critic network structure is introduced, and the delayed Actor network structure update frequency strategy is adopted.

[0217] The multi-UAV collaborative confrontation decision method based on a dual evaluation network provided in the embodiment of the present application can be executed by a multi-UAV collaborative confrontation decision device based on a dual evaluation network. In the embodiment of the present application, the multi-UAV collaborative confrontation decision method based on a dual evaluation network is executed by a multi-UAV collaborative confrontation decision device based on a dual evaluation network as an example to illustrate the multi-UAV collaborative confrontation decision method based on a dual evaluation network provided in the embodiment of the present application.

[0218] The embodiment of the present application also provides a multi-UAV collaborative confrontation decision-making device based on a dual evaluation network, which is applied to a target UAV in a confrontation training between a first UAV group and a second UAV group.

[0219] like Figure 5 As shown, the multi-UAV collaborative confrontation decision-making device based on the dual evaluation network includes: an acquisition module 510, a first processing module 520 and a second processing module 530.

[0220] An acquisition module 510 is used to acquire a first local observation state and action information of the target UAV performing a first action;

[0221] A first processing module 520 is used to input the first local observation state and the action information into a dual evaluation network to obtain a joint action output by the dual evaluation network, wherein the dual evaluation network includes: a current Actor network, a target Actor network, a first current Critic network, a second current Critic network, and a target Critic network;

[0222] The second processing module 530 is used to perform a cyclic calculation on the dual evaluation network based on the reward value corresponding to the joint action until the reward value converges, and determine the action parameters of the second action that the target drone needs to perform;

[0223] In the process of the double evaluation network cyclic calculation, the action value of the first action calculated by the first current Critic network and the second current Critic network is minimized to update the parameters of the current Actor network.

[0224] According to the multi-UAV collaborative confrontation decision-making device based on a dual evaluation network provided in an embodiment of the present application, by using a dual evaluation network obtained based on an improved Actor-Critic network structure in the target UAV during confrontation training, the action value calculated by the first current Critic network and the second current Critic network is minimized, thereby solving the problem that the action space where the action information of the training framework is located and the state space where the first local observation state is located are large, and can effectively reduce the impact of overestimation of the action value, and make a more accurate estimate of the action value of the UAV game, so that the target UAV chooses to execute the optimal second action according to the game strategy, thereby achieving fast and accurate decision-making in a changing battlefield environment, improving the success rate of combat mission execution, promoting the application of UAVs on future battlefields, and providing more efficient support for war operations.

[0225] The multi-UAV collaborative confrontation decision-making device based on a dual evaluation network in the embodiment of the present application is built into the target UAV.

[0226] The multi-UAV collaborative confrontation decision-making device based on the dual evaluation network provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented in the embodiment of the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network will not be repeated here.

[0227] In some embodiments, Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, each process of the embodiment of the multi-UAV collaborative confrontation decision-making method based on the dual evaluation network is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0228] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0229] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned multi-UAV collaborative confrontation decision-making method embodiment based on a dual evaluation network is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0230] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0231] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned multi-UAV collaborative confrontation decision-making method based on a dual evaluation network.

[0232] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0233] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned multi-UAV collaborative confrontation decision-making method embodiment based on a dual evaluation network, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0234] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0235] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0236] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network in each embodiment of the present application.

[0237] In the description of this application, "first feature" or "second feature" may include one or more of the features.

[0238] In the description of the present application, “plurality” means two or more.

[0239] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0240] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0241] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A multi-UAV collaborative confrontation decision-making method based on a dual evaluation network, characterized in that: The method is applied to a target drone for confrontation training between a first drone group and a second drone group, and comprises: Acquire a first local observation state and action information of the target UAV performing a first action; Inputting the first local observation state and the action information into a dual evaluation network to obtain a joint action output by the dual evaluation network, wherein the dual evaluation network includes: a current Actor network, a target Actor network, a first current Critic network, a second current Critic network, and a target Critic network; Based on the reward value corresponding to the joint action, the dual evaluation network is cyclically calculated until the reward value converges, and the action parameters of the second action that the target UAV needs to perform are determined; In the process of the double evaluation network cyclic calculation, the action value of the first action calculated by the first current Critic network and the second current Critic network is minimized to update the parameters of the current Actor network.

2. The multi-UAV collaborative confrontation decision-making method based on a dual evaluation network according to claim 1 is characterized in that: The step of taking a minimum value of the action value of the first action calculated by the first current critic network and the second current critic network includes: Where i is the number of the target drone; t is the time; k=1 represents the first current critic network, k=2 represents the second current critic network; y i,t is the minimum value of the action value of the first action of the target UAV i at time t, and the action value is used to update the parameters of the current Actor network; r i,t is the reward value of the target UAV i at time t; γ is the discount factor of the reward value; Q′ i is the target Critic network of the target UAV i; s i,t is the first local observation state of the target UAV i at time t; a i,t is the action information of the target UAV i at time t; θ i,k,Q′ is the network parameter of the target Critic network of the target UAV i; μ′ is the target Actor network of the target UAV i; θ i,k,μ′ are the network parameters of the target Actor network of the target UAV i; The current Actor network and the target Actor network have the same network structure, and the first current Critic network, the second current Critic network, and the target Critic network have the same network structure.

3. The multi-UAV collaborative confrontation decision-making method based on a dual evaluation network according to claim 1 is characterized in that: After determining the action parameters of the second action that the target UAV needs to perform, the method further includes: When the target UAV performs the second action based on the action parameter, obtaining a second local observation state corresponding to the second action; Taking a combination of the first local observation state, the action information, the reward value, and the second local observation state as a training sample, and storing the training sample in an experience replay pool; When the number of the training samples in the experience replay pool is greater than a quantity threshold, the training samples are extracted from the experience replay pool to update the parameters of the dual evaluation network.

4. The multi-UAV cooperative confrontation decision-making method based on a dual evaluation network according to claim 1 is characterized in that: The motion information is a motion variable determined based on the maneuvering motion of the target UAV in the maneuvering mode.

5. The multi-UAV cooperative confrontation decision-making method based on a dual evaluation network according to claim 4 is characterized in that: The maneuvers include constant speed level flight, accelerated level flight, decelerated level flight, fastest climb, left turn, right turn and dive.

6. The multi-UAV cooperative confrontation decision-making method based on a dual evaluation network according to any one of claims 1 to 5, characterized in that: The reward value is calculated based on the following: Among them, r i,t A is the reward value of the target UAV i at time t; i,t is the joint action of the target UAV i at time t; when the target UAV i belongs to the first UAV group, A i,t ∈t1 is used to indicate that the target UAV i is eliminated by the UAVs in the second UAV group at time t, A i,t ∈t2 is used to indicate that the missile launched by the target UAV i at time t has entered the state of being intercepted by the seeker, A i,t ∈t3 is used to represent that the target UAV i launches a missile when the UAV in the second UAV group enters the attackable range of the first UAV group at time t, A i,t ∈t4 is used to characterize that the rate of change of the distance between the target UAV i and the UAV j in the second UAV group at time t is a positive number.

7. The multi-UAV cooperative confrontation decision-making method based on a dual evaluation network according to any one of claims 1 to 5, characterized in that: The target critic network is used to evaluate the optimal game action.

8. The multi-UAV cooperative confrontation decision-making method based on a dual evaluation network according to any one of claims 1 to 5, characterized in that: The step of performing cyclic calculation on the dual evaluation network based on the reward value corresponding to the joint action includes: Substituting the first local observation state and the action information into the dual evaluation network, and updating the parameters of the current Actor network and the target Actor network according to the policy gradient.

9. A multi-UAV cooperative confrontation decision-making device based on a dual evaluation network, characterized in that: The device is applied to a target drone for confrontation training between a first drone group and a second drone group, and comprises: An acquisition module, used to acquire a first local observation state and action information of the target UAV performing a first action; A first processing module, configured to input the first local observation state and the action information into a dual evaluation network to obtain a joint action output by the dual evaluation network, wherein the dual evaluation network includes: a current Actor network, a target Actor network, a first current Critic network, a second current Critic network, and a target Critic network; A second processing module is used to perform a cyclic calculation on the dual evaluation network based on the reward value corresponding to the joint action until the reward value converges, and determine the action parameters of the second action that the target drone needs to perform; In the process of the double evaluation network cyclic calculation, the action value of the first action calculated by the first current Critic network and the second current Critic network is minimized to update the parameters of the current Actor network.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multi-UAV collaborative confrontation decision-making method based on a dual evaluation network as described in any one of claims 1 to 8 is implemented.

Images