Multi-unmanned aerial vehicle close-range fighting method based on opponent coding network
By introducing an opponent-encoding network, combining drone historical action information and mission environment information, optimizing the leadership network and opponent-encoding network, the complex computing and unstable environment of traditional drone close-range fighting algorithms in multi-unmanned airport scenes is solved. The sexual problem has achieved more efficient and stable multi-drone confrontation strategy generation.
Patent Information
- Application Number
- CN202411497544.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-02-25
AI Technical Summary
Traditional drone close-range fighting algorithms have problems such as complex calculations, poor stability and strong subjectivity in multiple unmanned airport scenarios, and are particularly difficult to effectively solve the environmental instability caused by the collaboration and confrontation of multiple agents.
Introduce an opponent-coded network, input the historical action information of the drone into the opponent-coded network, obtain the opponent-coded information through the center average calculation, and input it into the leadership network in combination with the task environment information to obtain instructions and guide evaluation networks and action networks. Training, optimize leadership networks and adversary coding networks.
It reduces the labor cost of drone behavior modeling, solves the problem that traditional algorithms cannot be applied to sequential decision-making tasks of multiple drones, and alleviates the environmental instability problem in the training process of multiple drones.
Smart Images
Figure CN119512141A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a UAV close combat method based on an opponent coding network, and in particular to a method for close combat in a multi-to-multi UAV scenario by introducing an opponent coding network into an action-evaluation algorithm, and belongs to the technical field of intelligent decision-making, intelligent task planning and intelligent command and control of military information systems. Background Art
[0002] With the development of science and technology, the pace of modern warfare is accelerating and the complexity is increasing. Artificial intelligence-assisted battlefield decision-making will play an increasingly important role. As the focal battlefield of future wars, the air combat field urgently needs rapid, automatic and autonomous decision-making to adapt to the high-speed, complex and changeable air battlefield environment.
[0003] With the rapid development of drone technology and its practical application, more and more researchers have begun to study drone close-range combat. As a typical confrontation scenario in future air combat battlefields, drone close-range combat has great advantages over manned fighters in terms of maneuverability, payload, and affordable cost. However, it is still far from meeting the requirements of fighter close-range combat in terms of target recognition, path selection, and battlefield judgment. Drone close-range combat urgently needs the support of intelligent air combat algorithms.
[0004] Traditional drone confrontation strategies based on game theory, influence diagrams, and decision tree methods always have a series of problems, such as complex calculations, poor stability, and strong subjectivity. Especially when there are a large number of drones involved in the fight, the amount of training and computational complexity of the algorithm model increase exponentially. How to efficiently and stably generate multi-drone confrontation strategies and optimize drone close-range combat technology in highly dynamic, complex, and uncertain air combat scenarios has put forward higher requirements for intelligent air combat algorithms.
[0005] Deep reinforcement learning, as a machine learning method that has been widely popular in recent years, has unique potential and advantages in solving the problem of sequential decision-making of single agents. Deep reinforcement learning abandons the shortcomings of traditional combat behavior modeling methods, relies on internal driving data for training, and allows agents to compete, train, and learn autonomously. There are no artificial preset rules, which is more in line with the learning laws of agents, and the training results are more convincing.
[0006] However, there are two difficulties that need to be solved in solving the multi-UAV close combat task. First, the output form of the network model in the traditional algorithm is only suitable for solving the single-agent confrontation task, and cannot effectively solve the environmental instability caused by the multi-agent collaborative confrontation; second, the traditional algorithm only considers the problem of self-strategy modeling, and does not model the environment and opponent's strategy, which reduces the accuracy and reliability of the strategy.
[0007] Therefore, it is necessary to propose a solution to improve one or more problems existing in the above-mentioned related technical solutions.
[0008] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0009] The present disclosure provides a method for close combat of a drone based on an opponent coding network, comprising the following steps:
[0010] According to the mission environment of close combat of multiple UAVs, mission environment information of the mission environment at each moment is obtained;
[0011] Initialize the opponent encoding network, the leadership network, the evaluation network and the action network; wherein each of the drones corresponds to an evaluation network and an action network;
[0012] According to the mission environment of the multi-UAV close combat, a mission is randomly collected;
[0013] Inputting the historical action information of each drone collection task into the opponent coding network to perform center average calculation to obtain the opponent coding information;
[0014] The mission environment information sensed by each drone and the opponent coding information are combined in parallel and input into the leadership network to obtain instruction information;
[0015] The task environment information and the instruction information sensed by each drone are combined and input into the evaluation network in parallel to obtain evaluation information, the action network outputs an action according to the evaluation information, and each drone performs an action according to the output action;
[0016] According to the gradient back propagation rule, the leadership network, the opponent encoding network, the evaluation network and the action network are optimized;
[0017] Returning to the task environment according to the multi-UAV close combat, randomly collecting steps of a task until the leader network and the opponent encoding network converge, and obtaining a convergence parameter;
[0018] The converged parameters are deployed to generate a multi-UAV close combat strategy.
[0019] In an exemplary embodiment of the present disclosure, each of the drones obtains the mission environment information and environment feedback at the next moment after taking action according to the mission environment information.
[0020] In an exemplary embodiment of the present disclosure, the historical action information includes historical movements and environmental feedback.
[0021] In an exemplary embodiment of the present disclosure, the center average is calculated as:
[0022]
[0023] The coded information k(a,r) represents an array with a storage capacity of 2, and x(a,r) represents all historical action information {(a1,r1),...,(a t-1 ,r t-1 )} in two-dimensional coordinates.
[0024] In an exemplary embodiment of the present disclosure, the expression for the parallel combination is:
[0025] s t Θk(a,r)
[0026] Among them, s t is the mission environment information perceived by each drone, k(a,r) is the opponent encoding information, Θ represents the combination of k(a,r) and s t Direct parallel connection.
[0027] In an exemplary embodiment of the present disclosure, the indication information is calculated using a non-linear activation function RELU.
[0028] In an exemplary embodiment of the present disclosure, the calculation method of combining the mission environment information and the instruction information perceived by each drone is the element cross multiplication method, and the expression is:
[0029] C input =s t ⊙Q(s,a,k)
[0030] Among them, Q(s,a,k) is the indication information s t ,s t is the mission environment information perceived by each UAV, C input Represents the input of the evaluation network C, and ⊙ represents the cross-multiplication of each dimension of the instruction information and each dimension of the task environment information.
[0031] The present invention discloses a method for close-range combat of unmanned aerial vehicles based on an opponent coding network, which can include the following beneficial effects: on the one hand, compared with the traditional expert system method, it avoids the modeling of close-range confrontation rules for multiple unmanned aerial vehicles and reduces the labor cost of modeling the behavior of unmanned aerial vehicles; on the other hand, by introducing an opponent coding network, inputting the historical action information of unmanned aerial vehicles and outputting the opponent coding information, it provides information assistance for the leadership network, guides the training of the lower-level evaluation network and the action network, solves the problem that traditional algorithms cannot be applied to the sequential decision-making tasks of multiple unmanned aerial vehicles, and at the same time, alleviates the problem of environmental instability in the training process of multiple unmanned aerial vehicles. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0033] Figure 1 A schematic diagram showing the steps of a drone close combat method based on an opponent coding network in an exemplary embodiment of the present disclosure;
[0034] Figure 2 A schematic diagram of an algorithm framework based on an opponent coding network in an exemplary embodiment of the present disclosure is shown;
[0035] Figure 3 A schematic diagram showing the interaction between the algorithm model based on the opponent coding network and the simulation environment in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0037] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0038] This example embodiment provides a method for close combat of drones based on an opponent coding network, such as Figure 1 As shown, the following steps may be included:
[0039] Step S101: according to the task environment of close combat of multiple UAVs, obtaining the task environment information of the task environment at each moment;
[0040] Step S102: Initializing the opponent encoding network, the leader network, the evaluation network and the action network; wherein each of the drones corresponds to an evaluation network and an action network;
[0041] Step S103: randomly collecting a task according to the task environment of close combat between multiple UAVs;
[0042] Step S104: inputting the historical action information of each drone collection task into the opponent coding network to perform center average calculation to obtain the opponent coding information;
[0043] Step S105: The mission environment information and opponent coding information sensed by each drone are combined in parallel and input into the leadership network to obtain instruction information;
[0044] Step S106: combining the mission environment information and the instruction information sensed by each drone and inputting them into the evaluation network to obtain evaluation information, the action network outputs an action according to the evaluation information, and each drone performs an action according to the output action;
[0045] Step S107: Optimize the leader network, opponent encoding network, evaluation network and action network according to the gradient backpropagation rule;
[0046] Step S108: Return to the task environment according to the multi-UAV close combat, randomly collect the steps of a task until the leader network and the opponent encoding network converge, and obtain the convergence parameters;
[0047] Step S109: deploying the convergence parameters to generate a multi-UAV close combat strategy.
[0048] In the disclosed embodiment, a method for close combat of unmanned aerial vehicles based on an opponent coding network is proposed, which can include the following beneficial effects: on the one hand, compared with the traditional expert system method, it avoids the modeling of close combat rules for multiple unmanned aerial vehicles and reduces the labor cost of unmanned aerial vehicle behavior modeling; on the other hand, by introducing the opponent coding network, inputting the historical action information of the unmanned aerial vehicle and outputting the opponent coding information, information assistance is provided to the leadership network, and the training of the lower-level evaluation network and the action network is guided, which solves the problem that the traditional algorithm cannot be applied to the sequential decision-making tasks of multiple unmanned aerial vehicles. At the same time, it alleviates the problem of environmental instability in the training process of multiple unmanned aerial vehicles.
[0049] It should be understood that when traditional algorithms solve multi-UAV close combat tasks, the algorithm output can only provide one UAV action. If a separate algorithm support is provided for each UAV, there is a problem of unstable training environment. The present invention proposes an opponent coding network based on the traditional algorithm framework, which can effectively solve the problem of multi-UAV collaborative confrontation without excessively increasing the computational complexity of the original algorithm. The core idea of the algorithm based on the opponent coding network is to design an action-evaluation network for each UAV. In addition, all UAVs innovatively share a leader network (LeaderNetwork) and an opponent coding network (Opponent-Coded Network), where the input of the Leader Network is the current situation and the opponent's coding information, and for each UAV, during the training process, it continuously receives the evaluation output of the LeaderNetwork.
[0050] The following is a more detailed description of each step of the above method in this exemplary embodiment.
[0051] Taking the generation of strategies for close combat between multiple drones in a battlefield environment as an example, this paper describes the generation and optimization process of multi-drone confrontation strategies based on the opponent encoding network. Figure 1-3 shown.
[0052] Step S101: according to the task environment of close combat of multiple UAVs, obtaining the task environment information of the task environment at each moment;
[0053] In the disclosed embodiment, a simulation environment of three drones in close combat with three drones is initialized, and the environmental information s t The dimension size is set to 100, and the dimension size of the drone action state is set to 5.
[0054] Step S102: Initializing the opponent encoding network, the leader network, the evaluation network and the action network; wherein each of the drones corresponds to an evaluation network and an action network;
[0055] In the disclosed embodiment, a leader network L is initialized: a fully connected neural network with 2 hidden layers (64 neurons) is used; an opponent encoding network O is defined, and the specific calculation method is the center average method:
[0056]
[0057] The coded information k(a,r) represents an array with a storage capacity of 2, and x(a,r) represents all historical action information {(a1,r1),...,(a t-1 ,r t-1 )} in two-dimensional coordinates, calculated as:
[0058]
[0059] For each drone i = {1, 2, 3}, an evaluation network C and an action network A are initialized: both use a fully connected neural network with 1 hidden layer (16 neurons).
[0060] Step S103: randomly collecting a task according to the task environment of close combat between multiple UAVs;
[0061] In the disclosed embodiment, a task T is randomly sampled from a multi-UAV close combat task simulation environment.
[0062] Step S104: inputting the historical action information of each drone collection task into the opponent coding network to perform center average calculation to obtain the opponent coding information;
[0063] In the disclosed embodiment, the historical action-reward {(a1, r1),..., (a t-1 ,r t-1 )} information is input to the opponent encoding network O for calculation, and the specific calculation method is the center average method:
[0064]
[0065] The coded information k(a,r) represents an array with a storage capacity of 2, and x(a,r) represents all historical action information {(a1,r1),...,(a t-1 ,r t-1 )} in two-dimensional coordinates, calculated as:
[0066]
[0067] The opponent encoding network O outputs the opponent encoding information k(a,r).
[0068] Step S105: The mission environment information and opponent coding information sensed by each drone are combined in parallel and input into the leadership network to obtain instruction information;
[0069] In the embodiment of the present disclosure, the mission environment information s sensed by each drone is t Combined with the opponent's encoded information k(a,r), the combination method is:
[0070] s t Θk(a,r)
[0071] Among them, Θ represents k(a,r) and environmental information s t Directly connected in parallel, that is, the dimension of environmental information is expanded from 100x1 to 102x1, N is the environmental information s tdimensionality and are input into the leadership network L in parallel.
[0072] Step S106: combining the mission environment information and the instruction information sensed by each drone and inputting them into the evaluation network to obtain evaluation information, the action network outputs an action according to the evaluation information, and each drone performs an action according to the output action;
[0073] In the embodiment of the present disclosure, the leader network L outputs instruction information Q(s, k) for each drone, and Q(s, k) is calculated as follows: the nonlinear activation function RELU
[0074] For each drone, the instruction information Q(s,a,k) and the mission environment information s perceived by the drone t Combined, the calculation method is element-wise cross multiplication:
[0075] C input =s t Q(s,a,k)
[0076] Among them, C input represents the input of the evaluation network C, represents the per-dimensional data and environmental information s in Q(s,a,k) t Each dimension of data is cross-multiplied, that is, 100x1 is expanded to 100x3, 100 is the environmental information s t , 3 is the dimension of the indication information Q(s,a,k), and they are input into the evaluation network C in parallel.
[0077] Evaluation network C outputs evaluation information Q(s,a,k) to action network A. For each drone, action network A outputs action a t , the drone performs action a t .
[0078] Step S107: According to the gradient back propagation rule, the leader network, the opponent encoding network, the evaluation network and the action network are optimized.
[0079] Step S108: Return to the task environment of close combat between multiple UAVs, randomly collect the steps of a task until the leader network and the opponent encoding network converge, and obtain the convergence parameters.
[0080] In the disclosed embodiment, if the leader network L and the opponent encoding network O have not converged, the process returns to step S103; otherwise, the process proceeds to step S109.
[0081] Step S109: deploying the convergence parameters to generate a multi-UAV close combat strategy.
[0082] In the disclosed embodiment, after convergence, the parameters of the leader network L, the opponent encoding network O, the evaluation network C, and the action network A are obtained, and the converged parameters are deployed in a simulation environment to generate a multi-UAV close combat strategy.
[0083] The core idea of the opponent coding network is to design an action-evaluation network for each drone. In addition, all drones share a leader network (LeaderNetwork) and an opponent coding network (Opponent-CodedNetwork), which inputs the drone's historical action information and outputs the opponent coding information to provide information assistance for the leader network and guide the training of the lower evaluation network and action network. This solves the problem that traditional algorithms cannot be applied to multi-drone sequential decision-making tasks. At the same time, it alleviates the problem of environmental instability in the multi-drone training process.
[0084] It should be noted that although several units of the system for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more units described above can be concretized in one unit. Conversely, the features and functions of a unit described above can be further divided into multiple units for concretization. Some or all of the units can be selected according to actual needs to achieve the purpose of the disclosed solution. Those of ordinary skill in the art can understand and implement it without paying creative work.
[0085] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. A multi-UAV close combat method based on an opponent coding network, characterized in that: The following steps are involved: According to the mission environment of close combat of multiple UAVs, mission environment information of the mission environment at each moment is obtained; Initialize the opponent encoding network, the leadership network, the evaluation network and the action network; wherein each of the drones corresponds to an evaluation network and an action network; According to the mission environment of the multi-UAV close combat, a mission is randomly collected; Inputting the historical action information of each drone collection task into the opponent coding network to perform center average calculation to obtain the opponent coding information; The mission environment information sensed by each drone and the opponent coding information are combined in parallel and input into the leadership network to obtain instruction information; The mission environment information sensed by each drone and the instruction information are combined and inputted into the evaluation network in parallel to obtain evaluation information, the action network outputs an action according to the evaluation information, and each drone performs an action according to the output action; According to the gradient back propagation rule, the leadership network, the opponent encoding network, the evaluation network and the action network are optimized; Returning to the task environment according to the multi-UAV close combat, randomly collecting steps of a task until the leader network and the opponent encoding network converge, and obtaining a convergence parameter; The converged parameters are deployed to generate a multi-UAV close combat strategy.
2. The multi-UAV close combat method based on the opponent coding network according to claim 1 is characterized in that: After each of the drones performs an action according to the mission environment information, it obtains the mission environment information and environment feedback at the next moment.
3. The multi-UAV close combat method based on the opponent coding network according to claim 2 is characterized in that: The historical action information includes historical actions and environmental feedback.
4. The multi-UAV close combat method based on the opponent coding network according to claim 1 is characterized in that: The central average is calculated as: The coded information k(a,r) represents an array with a storage capacity of 2, and x(a,r) represents all historical action information {(a1,r1),...,(a t-1 ,r t-1 )} in two-dimensional coordinates.
5. The method for close combat of multiple UAVs based on opponent coding network according to claim 4 is characterized in that: The expression of the parallel combination is: s t Θk(a,r) Among them, s t is the mission environment information perceived by each drone, k(a,r) is the opponent encoding information, Θ represents the combination of k(a,r) and s t Direct parallel connection.
6. The method for close combat of multiple UAVs based on opponent coding network according to claim 5 is characterized in that: The indication information is calculated using a nonlinear activation function RELU.
7. The method for close combat of multiple UAVs based on opponent coding network according to claim 6 is characterized in that: The calculation method of combining the mission environment information and the instruction information perceived by each drone is the element cross multiplication method, and the expression is: C input =s t ⊙Q(s,a,k) Among them, Q(s,a,k) is the indication information s t ,s t is the mission environment information perceived by each UAV, C input Represents the input of the evaluation network C, and ⊙ represents the cross-multiplication of each dimension of the instruction information and each dimension of the task environment information.