Unmanned aerial vehicle autonomous maneuver decision method and device based on space-time situation difference

Abstract

The application provides a kind of unmanned aerial vehicle autonomous maneuvering decision method and device based on space-time situation difference, wherein the above-mentioned method includes: obtaining the first space position of the first unmanned aerial vehicle after executing autonomous maneuvering decision and the second space position of the second unmanned aerial vehicle;Determine the first shortest distance of the first unmanned aerial vehicle to the second unmanned aerial vehicle attack area distance at each time step, and the second shortest distance of the second unmanned aerial vehicle to the first unmanned aerial vehicle attack area;Based on the first shortest distance and the second shortest distance, determine the relative attitude value between the first unmanned aerial vehicle and the second unmanned aerial vehicle;Determine the current reward feedback score of the first unmanned aerial vehicle at the current time step, and the historical reward feedback score of the first unmanned aerial vehicle at the last time step;Based on the current reward feedback score and the historical reward feedback score, determine the space-time situation difference reward function of the first unmanned aerial vehicle, wherein the space-time situation difference reward function is used to positively feedback or negatively feedback autonomous maneuvering decision.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology for unmanned systems, and in particular to an autonomous maneuvering decision-making method and device for unmanned aerial vehicles based on spatiotemporal situational difference. Background Technology

[0002] Maneuver decision-making is a crucial aspect of achieving autonomy for unmanned aerial vehicles (UAVs). It refers to the rapid and effective selection of maneuvers based on real-time situational information, and its quality directly reflects the UAV's level of intelligence.

[0003] Most current UAV maneuver decision-making methods based on deep reinforcement learning assume that the adversarial environment is a two-dimensional space and have relatively strict constraints on the initial state. They also often discretize the action space, resulting in poor generalization and versatility of the trained models.

[0004] This shows that the UAV maneuvering decision-making methods in related technologies have technical problems that make it difficult to meet the needs of real-world environments. Summary of the Invention

[0005] This invention provides an autonomous maneuvering decision-making method and apparatus for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference, which addresses the shortcomings of existing UAV maneuvering decision-making methods in meeting the needs of real-world environments, and enables more accurate reward feedback to UAVs.

[0006] This invention provides an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference, comprising the following steps. The process involves obtaining the first spatial position of a first UAV after executing an autonomous maneuver decision and the second spatial position of a second UAV, both located in a UAV adversarial simulation environment. Based on the first and second spatial positions, the process determines the first shortest distance between the first UAV and the attack zone of the second UAV at each time step, and the second shortest distance between the second UAV and the attack zone of the first UAV. Based on the first and second shortest distances, the process determines the relative attitude values ​​between the first and second UAVs. The process obtains the centroid distance between the first and second UAVs. The process determines the product of the relative attitude values ​​and correction coefficients, and the difference between the centroid distance and the maximum attack distance of the first UAV. The process uses the hyperbolic tangent function of the ratio of the product to the square of the difference as the reward feedback score of the first UAV. The process determines the current reward feedback score of the first UAV at the current time step and the historical reward feedback score of the first UAV at the previous time step. Based on the current reward feedback score and the historical reward feedback score, the process determines the spatiotemporal situation differential reward function of the first UAV, which is used to provide positive or negative feedback to the autonomous maneuver decision.

[0007] According to the present invention, an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference is provided. The step of determining the first shortest distance between the first UAV and the attack zone of the second UAV at each time step, based on the first spatial position and the second spatial position, includes: determining a first mass point of the first UAV and a second mass point of the second UAV based on the first spatial position and the second spatial position; determining the centroid distance between the first mass point and the second mass point; obtaining a first flight speed of the first UAV; determining the angle between the vector of the first flight speed and the vector of the centroid distance; when the second mass point is located in a region behind the first mass point and does not exceed a first region of the normal plane of the first flight speed, the centroid distance is taken as the first shortest distance between the second UAV and the attack zone of the first UAV; when the second mass point is located within the maximum attack range of the first mass point and not in a second region of the attack cone of the first UAV, the first shortest distance between the second UAV and the attack zone of the first UAV is determined based on the centroid distance, the angle, and the maximum attack angle of the first UAV.

[0008] According to the present invention, an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference is provided. Before obtaining the first spatial position of the first UAV after executing the autonomous maneuvering decision and the second spatial position of the second UAV, the method further includes: obtaining the spatial position, flight speed, pitch angle, and heading angle of each UAV in the ground coordinate system; determining the state space of each UAV based on the spatial position, flight speed, pitch angle, and heading angle; determining the action space of each UAV based on the tangential overload, normal overload, and velocity rotation angle of each UAV; determining the dynamic equation of each UAV based on the time-varying values ​​of the flight speed, pitch angle, and heading angle; determining the kinematic equation of each UAV based on the time-varying values ​​of the spatial position; and constructing a UAV adversarial simulation environment based on the state space, action space, dynamic equation, kinematic equation, and preset end conditions.

[0009] According to the present invention, an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference is provided. After constructing a UAV adversarial simulation environment based on the state space, the action space, the dynamic equations, the kinematic equations, and preset end conditions, the method further includes: obtaining the heading angle, pitch angle, and roll angle of a first UAV and a second UAV in the three-dimensional space of the UAV adversarial simulation environment, respectively; determining a first rotation matrix of the first UAV and a second rotation matrix of the second UAV based on the heading angle, the pitch angle, and the roll angle; determining the relative coordinates of the second UAV in the first body coordinate system of the first UAV; converting the second rotation matrix into a transformed second rotation matrix in the first body coordinate system based on the relative coordinates; and determining the state feature encoding between the first UAV and the second UAV based on the first rotation matrix and the transformed second rotation matrix.

[0010] According to the present invention, an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference is provided. After obtaining the first spatial position of the first UAV after executing the autonomous maneuvering decision and the second spatial position of the second UAV, the method further includes: initializing the execution policy network, execution value network, and experience buffer of a deep reinforcement learning decision-making model; storing the samples generated by the autonomous maneuvering decision into the experience buffer to obtain sampled experience data; optimizing the execution policy network and the execution value network based on the sampled experience data according to the target training steps to obtain the current execution network parameters; and updating the target policy network parameters and the target value network parameters based on the current execution network parameters to obtain the UAV game-theoretic decision-making model.

[0011] According to the present invention, an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference is provided. After updating the target policy network parameters and the target value network parameters based on the current execution network parameters to obtain the UAV game decision model, the method further includes: determining the target state feature encoding of the second UAV relative to the first UAV within the target time step; inputting the target state feature encoding into the UAV game decision model to obtain the target autonomous maneuvering decision for the first UAV output by the UAV game decision model.

[0012] This invention also provides an autonomous maneuvering decision-making device for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference, comprising the following modules: a first acquisition module, used to acquire a first spatial position of a first UAV after performing an autonomous maneuvering decision and a second spatial position of a second UAV, wherein the first UAV and the second UAV are located in a UAV adversarial simulation environment; a first determination module, used to determine, based on the first spatial position and the second spatial position, a first shortest distance of the first UAV relative to the attack zone of the second UAV and a second shortest distance of the second UAV relative to the attack zone of the first UAV at each time step; a second determination module, used to determine the relative attitude value between the first UAV and the second UAV based on the first shortest distance and the second shortest distance; and a second acquisition module, used to acquire... The system comprises: a centroid distance between the first UAV and the second UAV; a third determining module, configured to determine the product of the relative attitude value and the correction coefficient, and the difference between the centroid distance and the maximum attack range of the first UAV; a scoring module, configured to use the hyperbolic tangent function of the ratio of the product to the square of the difference as the reward feedback score of the first UAV; a fourth determining module, configured to determine the current reward feedback score of the first UAV at the current time step and the historical reward feedback score of the first UAV at the previous time step; and a fifth determining module, configured to determine the spatiotemporal situational differential reward function of the first UAV based on the current reward feedback score and the historical reward feedback score, wherein the spatiotemporal situational differential reward function is used to provide positive or negative feedback to the autonomous maneuver decision.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the UAV autonomous maneuvering decision-making method based on spatiotemporal situational difference as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the UAV autonomous maneuvering decision-making method based on spatiotemporal situational difference as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the UAV autonomous maneuvering decision-making method based on spatiotemporal situational difference as described above.

[0016] The present invention provides an autonomous maneuvering decision-making method and apparatus for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference. In an adversarial simulation environment, it acquires the first spatial position of a first UAV after executing an autonomous maneuvering decision and the second spatial position of a second UAV. Based on these two spatial positions, it determines the shortest distance between the attack zones of the first and second UAVs, thereby determining the relative attitude values ​​between them. This enables precise quantification of the spatiotemporal situation of both UAV adversaries. Based on the relative attitude values ​​and the distance between their centroids, it determines the reward feedback score of the first UAV and, based on its historical reward feedback scores, determines the spatiotemporal situational difference reward function. This introduction of temporal difference encourages UAVs to maintain a superior spatial situation or explore maneuvering strategies to turn disadvantage into advantage in adverse situations. This solves the technical problem that related UAV maneuvering decision-making methods often fail to meet the demands of real-world environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the autonomous maneuvering decision-making method for unmanned aerial vehicles based on spatiotemporal situational difference provided by the present invention.

[0019] Figure 2 These are simulation experiment effect diagrams provided by the present invention.

[0020] Figure 3 This is an overall flowchart of the UAV autonomous maneuver decision-making method based on spatiotemporal situational difference provided by the present invention.

[0021] Figure 4 This is a structural flowchart of the UAV autonomous maneuvering decision-making device based on spatiotemporal situational difference provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Compared to manned aircraft, drones possess advantages such as higher speed, greater overload capacity, more flexible maneuverability, and lower cost. Therefore, drones are widely used in various fields and play an irreplaceable role. With the development of information technology and artificial intelligence, the autonomous intelligence of drones is also continuously improving. Drone maneuver decision-making is a key aspect of achieving autonomous capabilities. Drone maneuver decision-making refers to the rapid selection of effective maneuvers based on real-time situational information, and its quality directly reflects the drone's level of intelligence.

[0025] Currently, UAV maneuver decision-making methods mainly fall into three categories: game theory-based methods, action planning-based methods, and machine learning-based methods. Game theory-based methods, including differential game theory and matrix game theory, have clear mathematical forms and rigorous reasoning processes, but their high solution complexity limits their suitability to simple maneuver decision-making problems. Action planning-based methods, including expert systems and influence graphs, simulate and analyze pilot decision-making logic, selecting a series of actions based on the situation and evaluating the optimal choice. However, their rules are rigid and difficult to adapt to complex and ever-changing battlefield environments. Machine learning-based methods, including genetic algorithms, neural networks, Bayesian networks, and reinforcement learning, fit the mapping relationship between situational information and decision sequences through data-driven approaches, continuously optimizing model structure and parameters, resulting in better adaptability to complex and changing environments. Deep reinforcement learning methods, which collect data and obtain rewards and penalties through UAV-environment interaction, yield more rational action decisions and possess long-term decision-making and planning capabilities, and have become a current research hotspot.

[0026] However, most current deep reinforcement learning-based UAV maneuver decision-making methods assume an adversarial environment in a two-dimensional space and impose strict constraints on the initial state. Furthermore, they often discretize the action space, resulting in poor generalization and versatility of the trained models, making it difficult to meet the demands of real-world environments. In addition, existing reinforcement learning methods suffer from high feature redundancy and fail to consider spatiotemporal trend information in their reward functions, leading to low data utilization and time-consuming training.

[0027] To address the aforementioned problems in the prior art, this invention provides an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference.

[0028] First, the UAV maneuver decision-making task is modeled as a Markov decision problem in a three-dimensional environment with continuous state space and continuous action space. The initial state of each game is randomly sampled in the state space to enhance the applicability of the model.

[0029] Secondly, this invention designs state features based on a rotation matrix, with opponent interaction information as the main component and individual information as a supplement, which effectively reduces information redundancy and improves the learning efficiency of the agent.

[0030] In addition, by combining expert knowledge, spatial geometric information, and changes in situational trends, the spatiotemporal situation of both sides in the UAV confrontation can be accurately quantified, providing the agent with more accurate reward feedback and improving the convergence speed of model training.

[0031] Finally, a historical trajectory modeling method is used to enhance the identification of the characteristic information of different opponent tactics. Then, a learning method based on adaptive opponent difficulty to adjust the training focus is combined, which greatly reduces the difficulty of model training and achieves the expected win rate while taking into account training efficiency.

[0032] refer to Figure 1 , Figure 1 This is a flowchart illustrating the UAV autonomous maneuver decision-making method based on spatiotemporal situational difference provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps.

[0033] Step 101: Obtain the first spatial position of the first UAV after executing autonomous maneuvering decision and the second spatial position of the second UAV, wherein the first UAV and the second UAV are located in a UAV adversarial simulation environment.

[0034] In drone-based combat maneuvering decision-making, one should try to keep one's own vulnerable rear areas away from the enemy's attack area, while simultaneously trying to ensure that one's own attack area covers the enemy's vulnerable rear areas.

[0035] In this embodiment of the invention, the UAV adversarial simulation environment includes the state space, action space, UAV kinematics and dynamics model, and end conditions for each of the multiple UAVs.

[0036] In this embodiment of the invention, the spatial position information of the first UAV after executing autonomous maneuvering decisions and the spatial position information of the second UAV are calculated and recorded in real time.

[0037] Step 102: Based on the first spatial position and the second spatial position, determine the first shortest distance between the first UAV and the attack zone of the second UAV at each time step, and the second shortest distance between the second UAV and the attack zone of the first UAV.

[0038] In an embodiment of the present invention, using and These represent the shortest distances from the location of the blue drone (i.e., the second drone mentioned above) to the attack area of ​​the red drone (i.e., the first drone mentioned above), and the shortest distances from the red drone to the attack area of ​​the blue drone, respectively. and Construct a spatial geometric situation function to reflect the situation information of the red and blue teams (i.e., the first UAV and the second UAV) at each time step (time point).

[0039] Step 103: Determine the relative attitude values ​​between the first UAV and the second UAV based on the first shortest distance and the second shortest distance.

[0040] In this embodiment of the invention, taking the first UAV (e.g., the aforementioned red UAV) as an example, the relationship between these two quantities (first shortest distance and second shortest distance) can reveal whether the red UAV is in an advantageous, disadvantageous, or balanced maneuvering posture.

[0041] Among them, the relative attitude value of the first UAV can be obtained through express, The range is [ , ], Used to indicate the maximum attack range of the first drone.

[0042] When the relative attitude value of the first drone When the value is greater than 0, the first UAV has a better maneuvering attitude than the second UAV. The larger the value, the greater the chance that the first drone will tail and chase the second drone, and vice versa.

[0043] Step 104: Obtain the centroid distance between the first UAV and the second UAV.

[0044] In this embodiment of the invention, in addition to the relative attitude value characterizing the degree of interaction between the player and the opponent in maneuvering posture, the distance between the centers of mass of the two sides (represented by D) is also one of the indicators for measuring the degree of interaction between the two sides in space.

[0045] Step 105: Determine the product of the relative attitude value and the correction coefficient, as well as the difference between the distance between the centroids of the two parties and the maximum attack range of the first UAV.

[0046] Step 106: The hyperbolic tangent function of the ratio of the product to the square of the difference is used as the reward feedback score for the first UAV.

[0047] Therefore, in this embodiment of the invention, based on the spatial geometric relationship formed by the red team's UAV (first UAV) in combat with the blue team's UAV (second UAV), the relative attitude value of the first UAV is used ( The function constructs the centroid distance (D) between the two drones to indicate the reward feedback score of the first drone at a certain moment (time step). For details, please refer to the following formula (1):

[0048] (1)

[0049] in, This represents the reward feedback score (function). This represents the hyperbolic tangent function (activation function). This represents the correction factor. This represents the relative attitude value of the first UAV. Indicates the distance between the centers of mass of the two sides. This indicates the maximum attack range of the first drone.

[0050] In an embodiment of the present invention, This represents the attack range constant for the drone. To correct the coefficient, and to encourage drones to eliminate opponents near their attack range, when hour, Set it to 10; otherwise set it to 5. This describes the score of the red team's UAV (first UAV) in a confrontation with the blue team's UAV (second UAV) at different distances and maneuvers at a certain moment. When the relative attitude values... When the distance D between the two sides is smaller, then The larger the value, the stronger the influence of the red drone on the blue drone, and vice versa. When the distance D between the two sides is constant, the relative attitude value... The larger the value, the better. The larger the value, the easier it is for the red team's drones to put the blue team within its attack range, and vice versa.

[0051] Step 107: Determine the current reward feedback score of the first drone at the current time step and the historical reward feedback score of the first drone at the previous time step.

[0052] Step 108: Based on the current reward feedback score and the historical reward feedback score, determine the spatiotemporal situational differential reward function of the first UAV, wherein the spatiotemporal situational differential reward function is used to provide positive or negative feedback for autonomous maneuvering decisions.

[0053] Reward feedback function It can only depict the real-time static spatial geometric state information of the first and second UAVs, but does not depict the trend information that changes over time. That is, the reward feedback function (score) is unrelated to the spatial geometric state of the opposing parties at the previous moment or the quality of the maneuver decisions made. In order to encourage the intelligent agent to maintain a better spatial state, or to explore a maneuver strategy to turn the disadvantage into an advantage in a disadvantageous situation, this embodiment of the invention introduces the concept of temporal difference to construct a spatiotemporal situational difference reward function, which can be referred to in the following formula (2):

[0054] (2)

[0055] in, The spatiotemporal situational differential reward function is represented. and Let represent the reward scores of the red team's drone (the first drone) under the spatial geometric relationship at time t and t-1, respectively. This is the signal conditioning coefficient, for example, it is set to 0.5 by default in the experiment.

[0056] In an embodiment of the present invention, the first item ( The second term reflects the reward in the spatial geometry of the agent's current location; It reflects the changing trend of the agent's spatial state from the previous moment to the current moment in terms of time.

[0057] Through the embodiments of the present invention, the spatiotemporal situation differential reward function design is an effective signal feedback amplification mechanism. If the second part is positive, it means that the agent has explored excellent maneuver decisions, putting itself (the first UAV) in a better spatial geometric situation than the previous moment. Therefore, a positive feedback signal is given to encourage it. If the second part is negative, it means that the decision made by the agent is worse than the previous moment. Therefore, a negative feedback signal is given to punish it.

[0058] Through the steps described above in this embodiment of the invention, the first spatial position of the first UAV after executing autonomous maneuvering decisions and the second spatial position of the second UAV are obtained, wherein the first UAV and the second UAV are located in a UAV adversarial simulation environment; based on the first spatial position and the second spatial position, the first shortest distance of the first UAV relative to the attack zone of the second UAV and the second shortest distance of the second UAV relative to the attack zone of the first UAV are determined at each time step; based on the first shortest distance and the second shortest distance, the relative attitude values ​​between the first UAV and the second UAV are determined; the centroid distance between the first UAV and the second UAV is obtained; the product of the relative attitude value and the correction coefficient, and the centroid distance between the two UAVs and the maximum attack of the first UAV are determined. The difference in distance is used as the hyperbolic tangent function of the ratio of the product to the square of the difference as the reward feedback score of the first UAV. The current reward feedback score of the first UAV at the current time step is determined, along with the historical reward feedback score of the first UAV at the previous time step. Based on the current reward feedback score and the historical reward feedback score, the spatiotemporal situational difference reward function of the first UAV is determined, whereby the spatiotemporal situational difference reward function is used to provide positive or negative feedback for autonomous maneuver decisions. Thus, the introduction of the temporal difference concept can encourage the UAV to maintain a superior spatial situation or explore maneuver strategies to turn disadvantage into advantage in adverse situations. This solves the technical problem that UAV maneuver decision-making methods in related technologies are difficult to meet the needs of real-world environments.

[0059] According to the present invention, an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference is provided. Based on a first spatial position and a second spatial position, the method determines the first shortest distance between the first UAV and the attack zone of the second UAV at each time step, including:

[0060] Based on the first spatial position and the second spatial position, the first mass point of the first UAV and the second mass point of the second UAV are determined respectively.

[0061] Determine the distance between the centers of mass of the first and second particles;

[0062] Obtain the first flight speed of the first drone;

[0063] Determine the angle between the vector of the first flight velocity and the vector of the distance between the centers of mass of both sides;

[0064] When the second particle is located behind the first particle and does not exceed the first region of the normal plane of the first flight speed, the distance between the centers of mass of the two particles is taken as the shortest distance between the second UAV and the attack zone of the first UAV.

[0065] When the second mass is within the maximum attack range of the first mass, but not in the second region of the attack cone of the first drone, the shortest distance between the second drone and the attack zone of the first drone is determined based on the distance between the centers of mass of both parties, the included angle, and the maximum attack angle of the first drone.

[0066] In this embodiment of the invention, the shortest distance between the first UAV and the attack zone of the second UAV is determined ( Taking [example] as an example, we model the shortest distance from the position of the blue drone (second drone) to the attack zone of the red drone (first drone) under different conditions. Assume the point masses of the red and blue drones are […]. and The physical distance between the centers of mass of the two drones is The red team's speed is Maximum attack range is Its attack range in a three-dimensional plane is the spherical fan-shaped area, with a maximum attack angle of [missing information]. . The vector connecting the red and blue point masses (i.e., the distance between their centers of mass) The included angle between them can be referenced by the following formula (3):

[0067] (3)

[0068] in, The angle between the vector representing the first flight velocity and the vector representing the distance between the centers of mass of both sides. Indicates the first flight speed. A vector representing the distance between the centroids of the two sides.

[0069] The following describes how to determine the shortest distance for the blue team (second drone) to enter the red team's (first drone's) attack zone under different spatial positions.

[0070] like In the region behind r and not exceeding the area passing through point r The area of ​​the normal plane, then the blue side Distance from Red The shortest distance to the attack zone is: ,in, This indicates the distance between the centers of mass of the two sides.

[0071] like If the blue team is within r's maximum attack range but not within the attack zone and not within the aforementioned area, then... Distance from Red The shortest distance to the attack zone is: ,in, Distance between the centers of mass of both sides The angle between the vector representing the first flight velocity and the vector representing the distance between the centers of mass of both sides. This indicates the maximum angle of attack for the first drone.

[0072] like If the attack is outside the maximum attack range of r but not within the extended range of the attack cone and not within the aforementioned area, then the blue team... Distance from Red The shortest distance to the attack zone is: ,in , ,in, Indicates the distance between the centers of mass of the two sides. The angle between the vector representing the first flight velocity and the vector representing the distance between the centers of mass of both sides. This indicates the maximum angle of attack for the first drone.

[0073] like If the attack is outside the maximum attack range of r but within the extended range of the attack cone, then the blue team... Distance from Red The shortest distance to the attack zone is: ,in, Distance between the centers of mass of both sides This indicates the maximum attack range of the first drone.

[0074] Based on the above formula, embodiments of the present invention can determine the shortest distance for the blue team (second drone) to enter the red team's (first drone's) attack area at each time step. Conversely, it also allows calculation of the shortest distance for the red team to enter the blue team's attack zone. Taking the red drone as an example, the relationship between these two quantities reveals whether the red drone is in an advantageous, disadvantageous, or balanced maneuvering posture. The relative attitude values ​​between the red drone and the blue drone... The range is [ , ],when When the value is greater than 0, the red team's drone has a greater advantage in maneuverability compared to the blue team's. The larger the value, the greater the chance that the red side will catch up with the blue side, and vice versa.

[0075] According to the present invention, an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference is provided. Before obtaining the first spatial position of the first UAV after performing autonomous maneuvering decision and the second spatial position of the second UAV, the method further includes:

[0076] Obtain the spatial position, flight speed, pitch angle, and heading angle of each UAV in the ground coordinate system;

[0077] The state space of each UAV is determined based on its spatial position, flight speed, pitch angle, and heading angle.

[0078] The motion space of each UAV is determined based on its tangential overload, normal overload, and velocity rotation angle.

[0079] Based on the changes in flight speed, pitch angle, and heading angle over time, the dynamic equations for each UAV are determined.

[0080] Based on the change in spatial position over time, the kinematic equations for each UAV are determined;

[0081] A drone combat simulation environment is constructed based on state space, action space, dynamic equations, kinematic equations, and preset final conditions.

[0082] In this embodiment of the invention, the state space definition of the UAV can refer to the following formula (4):

[0083] (4)

[0084] in, Represents the state space of the drone. The three components representing the spatial position of the UAV in the ground coordinate system; Indicates the flight speed of the drone. Indicates the pitch angle of the drone. This indicates the heading angle (yaw angle), with the leftward yaw of the aircraft as positive.

[0085] The three-dimensional vector of the UAV's motion space is encoded and used as a control variable to control the UAV's motion space. For details, please refer to the following formula (5):

[0086] (5)

[0087] in, Indicates the action space of the drone. Tangential overload represents the ratio of the resultant force of thrust and drag on the UAV along the velocity direction to the UAV's own weight. Normal overload represents the ratio of the resultant force on the plane perpendicular to the velocity direction to the aircraft's own weight. The roll angle is the angle between the aircraft's plane of symmetry and the vertical plane containing the aircraft's flight path. Considering the physical properties of the UAV, the ranges of the three components are limited: the normal overload range is... Tangential overload range is The range of speed roll angle is .

[0088] Establish a three-degree-of-freedom dynamic equation in the UAV (aircraft) trajectory coordinate system, and refer to the following formula (6):

[0089] (6)

[0090] in, Indicates the drone's flight speed. Indicates a time step. Represents gravity. This indicates tangential overload of the drone. Indicates the pitch angle of the drone. This indicates the normal overload of the drone. For the drone's speed roll angle, This indicates the heading angle of the drone.

[0091] Numerical integration can be used to obtain the changes in the aircraft's velocity, pitch angle, and yaw angle over time after the maneuver. Then, the aircraft's kinematic equations are established in the ground coordinate system, as shown in the following formula (7):

[0092] (7)

[0093] in, The three components representing the spatial position of the UAV in the ground coordinate system. Indicates a time step. Indicates the drone's flight speed. Indicates the pitch angle of the drone. This indicates the heading angle of the drone.

[0094] The aircraft's position in three-dimensional space under the new state can be obtained by solving the differential equations. Both sides use the above dynamic and kinematic equations to calculate their new state information after performing their respective actions.

[0095] In this embodiment of the invention, the end conditions for drone combat are set to six scenarios: red wins, blue wins, draw, red goes out of bounds, blue goes out of bounds, and timeout. The rewards for the corresponding end states are 300, -100, -0, -100, 0, and 0, respectively. The specific end condition determination method is shown in the following formula (8):

[0096] (8)

[0097] in, Let the vector represent the spatial position of the red team (first UAV) pointing towards the blue team (second UAV). Let be the angle between the speed of the red team's drone and the distance between the centers of mass of both sides. Let θ be the angle between the speed of the blue drone and the distance between the centers of mass of both sides.

[0098] According to the present invention, an autonomous maneuver decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference is provided. After constructing a UAV adversarial simulation environment based on state space, action space, dynamic equations, kinematic equations, and preset final conditions, the method further includes:

[0099] Obtain the heading angle, pitch angle, and roll angle of the first and second UAVs in a three-dimensional space within a UAV combat simulation environment;

[0100] Based on the heading angle, pitch angle, and roll angle, the first rotation matrix of the first UAV and the second rotation matrix of the second UAV are determined respectively.

[0101] Determine the relative coordinates of the second UAV within the first body coordinate system of the first UAV;

[0102] Based on relative coordinates, the second rotation matrix is ​​transformed into a transformed second rotation matrix located in the first body coordinate system;

[0103] Based on the first rotation matrix and the transformed second rotation matrix, the state feature encoding between the first UAV and the second UAV is determined.

[0104] In three-dimensional space, the attitude angles of the UAV are the heading angle and the yaw angle. Pitch angle and roll angle These represent the angles of rotation of the aircraft around the x, y, and z axes, respectively. Next, the rotation matrices for each corresponding angle of rotation around each axis will be calculated. For details, please refer to the following formula (9):

[0105] (9)

[0106] Among them, in three-dimensional space ( The attitude angles of the UAV are the heading angles. Pitch angle and roll angle .

[0107] Calculate the rotation matrix of the aircraft from the body coordinate system to the inertial coordinate system according to the rotation order of the coordinate axes z, y, and x. For details, please refer to the following formula (10):

[0108] (10)

[0109] in, Indicates the heading angle of the drone. Indicates the pitch angle of the drone. This indicates the roll angle of the drone.

[0110] Substituting the corresponding Euler angles for the red team (the first drone), we obtain its rotation matrix as follows: Similarly, the rotation matrix of the blue team (the second drone) is: .

[0111] Assuming the red team's coordinates are located at the origin, and the blue team's coordinates need to be translated, then the blue team's relative coordinates are: You can refer to the following formula (11).

[0112] (11)

[0113] in, Indicates the relative coordinates of the blue team. These represent the x-coordinate, y-coordinate, and vertical coordinate, respectively. It represents the blue side. It represents the red side. This represents the transpose of the matrix.

[0114] Combined with the red side's rotation matrix The relative coordinates of the blue team in the red team's coordinate system can be obtained. For details, please refer to the following formula (12):

[0115] (12)

[0116] in, This represents the spatial rotation in three-dimensional space from the inertial coordinate system to the red-side machine coordinate system. Indicates the relative coordinates of the blue team. This represents the relative coordinates of the blue team in the red team's coordinate system. The coordinates in the red team's coordinate system are labeled as follows: =( , ,

[0117] Rotate the blue side's matrix Rotation matrix for transformation to the red team's coordinate system For details, please refer to the following formula (13):

[0118] (13)

[0119] in, This represents the rotation matrix of the blue side. This represents the spatial rotation in three-dimensional space from the inertial coordinate system to the red-side machine coordinate system. This represents the rotation matrix that transforms the blue team's rotation matrix to the red team's body coordinate system.

[0120] make The Middle Line number The elements of a column are denoted as The heading angle corresponding to the blue side in the red coordinate system can be obtained by inverse solving using the rotation matrix. Pitch angle and roll angle For details, please refer to the following formula (14):

[0121] (14)

[0122] In summary, by transforming from the inertial coordinate system to the red team's body coordinate system, the following state feature encoding representation of the red and blue teams in a UAV combat environment is constructed:

[0123]

[0124] The state representation has a dimension of 10 and consists of three parts: the first part ( Let be the altitude, velocity magnitude, and pitch angle of the red team in the absolute coordinate system. Here, the red team's attitude angles only concern the pitch angles related to the direction of gravity and the flight limit altitude. Our own heading angle This is irrelevant and therefore does not need to be considered; the second part is the relative speed between the blue and red sides. The third part is the blue coordinates in the red body coordinate system calculated based on the rotation matrix. , , ) and the blue team's attitude angle ( ).

[0125] In the red team's coordinate system, the red team's coordinates and attitude angles are both 0. The blue team's new coordinates and attitude angles are the opponent's state information perceived by the red team from its first-person perspective. This greatly enhances the agent's situational awareness and makes it easier for the decision-making algorithm to learn and deduce the relationship between states, thereby effectively reducing information redundancy and improving the agent's decision-making learning efficiency.

[0126] According to the present invention, an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference, after obtaining the first spatial position of the first UAV after performing autonomous maneuvering decision and the second spatial position of the second UAV, the method further includes:

[0127] Initialize the execution policy network, execution value network, and experience buffer of the deep reinforcement learning decision model;

[0128] The samples generated by autonomous maneuver decisions are stored in an experience buffer to obtain sampled experience data;

[0129] Based on the target number of training steps and sampled empirical data, the execution policy network and execution value network are optimized to obtain the current execution network parameters;

[0130] Based on the current execution network parameters, the target policy network parameters and target value network parameters are updated to obtain the UAV game decision model.

[0131] In this embodiment of the invention, the Twin Delayed Deep Deterministic policy gradient algorithm (TD3) is used to train the maneuver decision model. The policy network, value network, and experience buffer of the deep reinforcement learning decision model are initialized. Samples generated during the UAV's maneuver decision-making process are stored in the buffer, and experience data is sampled. Gradient descent is used to optimize the value network parameters. Every certain number of training steps, the policy network parameters are optimized using the policy gradient. The target policy network and target value network parameters are updated using the current network parameters. Specific training steps are detailed in the following steps.

[0132] Step 1: Initialize the parameters of the execution policy network respectively. Parameters of the target policy network Parameters of the two execution value networks and Parameters of the two target value networks and Experience buffer zone, motion noise And synchronize the parameters of each target network with the parameters of the corresponding execution network.

[0133] Step 2: Randomly initialize the states of both adversaries and preprocess them to obtain the overall state of the environment. .

[0134] Step 3, based on the overall environmental status Utilizing an execution strategy network with added noise to select actions After performing the action, the next state is obtained. and rewards .

[0135] Step 4, collect the data. Sample data is stored in a buffer.

[0136] Step 5: Sample empirical data in batches of N from the buffer. The gradient descent method is used to iteratively optimize the parameters of the execution value network based on the loss function.

[0137] The loss function can be found in the following formula (15):

[0138] (15)

[0139] in, Represents the loss function. Indicates the first Parameters of a Q-network Indicates the network selection action of the execution strategy. The reward Indicates the discount factor. This indicates the next state obtained after performing an action. Indicates the use of parameters After the target policy network executes an action, it obtains the next state. Indicates exploratory noise. Indicates the use of parameters The Q network in state action Q-value prediction under the following conditions This indicates the batch size, which is the number of samples used in a single update.

[0140] Step 6: Every certain number of training steps, optimize the parameters of the policy network using the policy gradient method.

[0141] For details, please refer to the following formula (16):

[0142] (16)

[0143] in, Represents the policy gradient. Indicates the state Below, regarding the action Q value Regarding the action gradient, The gradient is represented as... equal strategy The output action is calculated. Representation Strategy Regarding parameters The gradient is used to represent how changing the parameters of the policy affects the state. The action to choose from, This indicates the batch size, which is the number of samples used in a single update.

[0144] Step 7: If the current state is the final state, continue from step 2; if the current state is not the final state, then... As the current state, continue from step 3.

[0145] Step 8: When the training objective is achieved, save the target network as a model for drone game decision-making.

[0146] According to the present invention, an autonomous maneuvering decision-making method for unmanned aerial vehicles (UAVs) based on spatiotemporal situational difference, after updating the target policy network parameters and the target value network parameters based on the current execution network parameters to obtain the UAV game-theoretic decision-making model, the method further includes:

[0147] Determine the target state feature encoding of the second UAV relative to the first UAV within the target time step;

[0148] The target state feature encoding is input into the UAV game decision model to obtain the target autonomous maneuver decision for the first UAV output by the UAV game decision model.

[0149] In this embodiment of the invention, when the opponent's situation is uncertain, an adaptive opponent training method is proposed.

[0150] When the adversarial environment switches to different types of opponents against the red side, the different strategies employed by these opponents mean that the maneuver strategies previously learned by the agent for one type of opponent may not be applicable to other types. If the agent continues to be trained to adapt to the strategies of the new opponents, it may forget the strategies used against the previous opponent or experience training non-convergence.

[0151] To address situations with uncertain opponents, we propose an adaptive opponent training method that combines implicit opponent modeling with multi-task learning. First, we enhance the identification of different opponent tactics by modeling historical trajectories. Then, we set the reward in the adversarial environment as a reward function based on spatiotemporal situational differences. Finally, based on the adversarial performance, we dynamically adjust the sampling weight of each opponent in the opponent pool during training, achieving the expected win rate while maintaining training efficiency.

[0152] Specifically, the aforementioned feature information for enhancing the identification of different opponent tactics based on historical trajectory modeling can be: a state encoding method based on the transformation from absolute coordinate system to body coordinate system, which learns the opponent type representation for the opponent's historical trajectory of the previous k steps, and improves the ability to identify different types of opponents by learning the opponent's movement trend features. The specific steps can be referred to as follows:

[0153] Step 1: Initialize a global queue container que, set the queue size to L, and use it to store the opponent's historical state encoding information from the previous L steps.

[0154] Step 2: When the drone game simulation platform reaches the t-th time step, the original state encoding information is obtained. State coding method based on coordinate system transformation get .

[0155] in, This represents the state feature encoding after coordinate system transformation, which can be found in the above text and will not be repeated here.

[0156] Step 3: Determine the status code of the blue team.

[0157] The specific status coding for the blue team can be referenced in the following formula (17):

[0158] (17)

[0159] in, This represents the transformation from the inertial coordinate system of the blue unit to the coordinate system of the red unit, and the blue unit's coordinates in the red unit's coordinate system. , , ) and the blue team's attitude angle ( ), This indicates the speed change of the blue team's machine.

[0160] Step 4: Combining the opponent state encoding information from the first L time steps stored in the queue container, the final input to the policy network is formed. .

[0161] The final input can be referenced from the following formula (18):

[0162] = ( (18)

[0163] in, This indicates the final input. Represents state feature encoding, This indicates the status code of the blue team. Indicates a time step. This represents the time step queue.

[0164] Step 5, The input is fed into the current policy network to obtain the action. .

[0165] Step 6, perform the action Input into the simulation platform and receive a reward. and the original state encoding information for the next time step .

[0166] Step 7: Repeat steps 3 and 4 to process. ,get .

[0167] Step 8, finally we get the tuple. It is stored in the experience replay pool.

[0168] in, Represents status coding information, Indicates an action, Indicates a reward. This indicates the next state encoding information.

[0169] Embed the opponent's historical trajectory information into the state coding information. In this way, it can effectively improve the ability of intelligent agents to identify different types of opponents, and then output the optimal maneuvering action according to the different types of opponents to defeat them.

[0170] In this embodiment of the invention, the sampling weight of each opponent in the opponent pool is dynamically adjusted during the training process based on the adversarial performance.

[0171] Specifically, in order to improve the ability of intelligent agents (drones) to adapt to different opponent strategies, that is, to adapt to different tasks, a method is proposed that can dynamically switch the game task being trained during the training process. Specifically, the training framework is adjusted, the performance against each opponent is recorded during the training phase, the opponent pool is uniformly sampled before the start of each game to serve as the opponent for that game, and as training progresses, the sampling weight of each opponent in the opponent pool is automatically adjusted based on the performance against each opponent, with more difficult opponents receiving a larger weight and relatively easier opponents receiving a smaller weight.

[0172] This training method, which automatically adjusts the opponent's appearance ratio based on adversarial performance, can accelerate the learning convergence of the decision network. The specific implementation steps are as follows:

[0173] Step 1: Initialize N queues Q of size L to record the historical game results of N opponents.

[0174] Step 2: Randomly initialize a relative weight list W with a capacity of N.

[0175] Step 3: Before each match begins, the opponent is randomly selected based on W.

[0176] Step 4: After each match, the match score is updated to the corresponding opponent's match score queue. middle.

[0177] Step 5: After every M rounds, update W by calculating the non-win (opponent wins, out of bounds, timeout) rate in the game against each opponent.

[0178] Step 6: Repeat steps 3 through 5 until training is complete.

[0179] In summary, this invention models the UAV maneuver decision-making task as a Markov decision problem in a three-dimensional environment with continuous state and action spaces. Furthermore, it constructs a state feature encoding representation for the red and blue teams by transforming the absolute coordinate system and the body coordinate system. This representation primarily uses opponent interaction information and secondarily uses individual information, effectively reducing state information redundancy and improving agent sample learning efficiency. In addition, the initial state of each round is randomly sampled within the state space, enhancing the model's versatility.

[0180] This invention combines expert knowledge and spatial geometric information to quantify the spatial situation of both sides in a UAV confrontation, and introduces the concept of temporal difference to characterize the trend changes of the situation, thereby constructing a spatiotemporal situational difference reward function, providing UAVs with more accurate reward feedback, overcoming the reward sparsity problem, and improving the model's convergence speed and decision-making ability.

[0181] To address situations with uncertain opponents, this invention proposes an adaptive opponent training method that combines implicit opponent modeling and multi-task learning. It utilizes historical trajectory modeling to enhance the identification of feature information of different opponent tactics and dynamically adjusts the sampling weight of each opponent in the opponent pool during the training process based on the adversarial performance, which greatly reduces the difficulty of model training and achieves the expected win rate while also taking into account training efficiency.

[0182] The following describes the specific implementation of the UAV autonomous maneuver decision-making method based on spatiotemporal situational difference provided by the present invention in practical applications.

[0183] refer to Figure 2 , Figure 2 These are simulation experiment effect diagrams provided by the present invention.

[0184] In a simulated adversarial environment, the normal overload range of the action space is set to... Tangential overload range is The range of speed roll angle is The attack range was set to 400 meters, the gravitational acceleration to 9.8, and the time step to 1, meaning each action took 1 second to execute. In the training environment, the coordinates, velocities, and Euler angles of both sides in each game were initialized using uniform random sampling. The initialization ranges for both sides' states are shown in Table 1.

[0185] Table 1: Initialization Range of Both Parties' States

[0186]

[0187] The reinforcement learning algorithm TD3 was used with the following parameter settings: 10,000 initial training steps, 100,000 experience replay pool size, 100 batch size, a learning rate of 5e-4 for both the actor and evaluator networks, a Gaussian noise variance of 0.1 for reinforcement exploration, and a discount rate of 0.99. We used the red side's win rate as the evaluation metric, counting the number of games in the last 1000 where the red side won, then dividing by 1000 to convert it to a percentage.

[0188] This invention conducts experimental verification under uncertain opponent conditions. The opponent pool includes two types: constant linear strategy and Min-Max strategy. Using the coordinate system transformation state encoding and spatiotemporal situational differential reward function proposed in this invention, the results of different training methods are compared, including three training methods: opponent uniform sampling, opponent trajectory modeling + uniform sampling, and opponent trajectory modeling + adaptive sampling.

[0189] Table 2 shows the win rates of the red team against the blue team with different strategies under different training methods. The maximum training steps were set to 2.5 million. It can be seen that the training method with uniform sampling of opponents did not converge after reaching the maximum training steps, and its win rate against the Min-Max strategy was still relatively low. The opponent trajectory modeling + uniform sampling training method achieved a win rate of not less than 0.8 after 2.5 million steps, meaning the red team model could defeat all opponents in the opponent pool simultaneously. The opponent trajectory modeling + adaptive sampling training method converged at 1.5 million steps, achieving a win rate of not less than 0.84, meaning the red team model could defeat all opponents in the opponent pool simultaneously. Therefore, the method proposed in this invention can improve sample utilization, increase model convergence speed, and improve the accuracy of the model's strategy generation.

[0190] Table 2: Win rate of Red against Blue with different strategies under different training methods

[0191]

[0192] refer to Figure 3 , Figure 3 This is an overall flowchart of the UAV autonomous maneuver decision-making method based on spatiotemporal situational difference provided by the present invention. For details, please refer to the following steps.

[0193] Step 1: Model the UAV maneuver decision-making task as a Markov decision problem in a three-dimensional environment with continuous state space and continuous action space. Establish an adversarial simulation environment, including UAV kinematics and dynamics modeling, and define the state space, action space and final conditions.

[0194] Step 2: Randomly initialize the state information of both sides in each game in the state space to enhance the practicality of the model. Design a state encoding method based on coordinate system transformation, using state features that are mainly based on opponent interaction information and supplemented by individual information, which effectively reduces information redundancy and improves the learning efficiency of the agent.

[0195] Step 3: Design a reward function based on spatiotemporal situational difference. This function combines expert knowledge, spatial geometric information, and situational trend changes to accurately quantify the spatiotemporal situation of both sides in the UAV confrontation, providing the agent with more accurate reward feedback and improving the convergence speed of model training.

[0196] Step 4, train the maneuver decision model: initialize the policy network, value network and experience buffer of the deep reinforcement learning decision model, store the samples generated by the UAV maneuver decision process into the buffer and sample experience data, optimize the value network parameters using gradient descent, optimize the policy network parameters using policy gradient descent at regular intervals, and update the target policy network and target value network parameters with the current network parameters.

[0197] Step 5: For situations where the opponent is uncertain, an adaptive opponent training method is proposed. Based on historical trajectory modeling, the characteristic information of different opponent tactics is enhanced. Then, based on the adversarial performance, the sampling weight of each opponent in the opponent pool is dynamically adjusted during the training process, so as to achieve the expected win rate while taking into account the training efficiency.

[0198] Step 6: Repeat steps 3-5 until the final state is reached, using the target network as the model for UAV maneuvering decisions.

[0199] The following describes the UAV autonomous maneuvering decision-making device based on spatiotemporal situational difference provided by the present invention. The UAV autonomous maneuvering decision-making device based on spatiotemporal situational difference described below can be referred to in correspondence with the UAV autonomous maneuvering decision-making method based on spatiotemporal situational difference described above.

[0200] refer to Figure 4 , Figure 4 This is a structural flowchart of the UAV autonomous maneuver decision-making device based on spatiotemporal situational difference provided by the present invention, which includes: a first acquisition module 401, a first determination module 402, a second determination module 403, a second acquisition module 404, a third determination module 405, a scoring module 406, a fourth determination module 407, and a fifth determination module 408.

[0201] The first acquisition module 401 is used to acquire the first spatial position of the first UAV after executing autonomous maneuvering decision and the second spatial position of the second UAV, wherein the first UAV and the second UAV are located in a UAV adversarial simulation environment.

[0202] The first determining module 402 is used to determine, based on the first spatial position and the second spatial position, the first shortest distance of the first UAV to the attack zone of the second UAV at each time step, and the second shortest distance of the second UAV to the attack zone of the first UAV.

[0203] The second determining module 403 is used to determine the relative attitude values ​​between the first UAV and the second UAV based on the first shortest distance and the second shortest distance;

[0204] The second acquisition module 404 is used to acquire the centroid distance between the first UAV and the second UAV.

[0205] The third determining module 405 is used to determine the product of the relative attitude value and the correction coefficient, as well as the difference between the distance between the centroids of the two parties and the maximum attack range of the first UAV.

[0206] The scoring module 406 is used to use the hyperbolic tangent function of the ratio of the product to the square of the difference as the reward feedback score for the first UAV.

[0207] The fourth determination module 407 is used to determine the current reward feedback score of the first UAV at the current time step and the historical reward feedback score of the first UAV at the previous time step.

[0208] The fifth determining module 408 is used to determine the spatiotemporal situation differential reward function of the first UAV based on the current reward feedback score and the historical reward feedback score. The spatiotemporal situation differential reward function is used to provide positive or negative feedback for autonomous maneuver decisions.

[0209] Specifically, the UAV autonomous maneuvering decision-making device based on spatiotemporal situational difference provided by the present invention can realize all the method steps implemented in the above-mentioned UAV autonomous maneuvering decision-making method embodiment based on spatiotemporal situational difference, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0210] refer to Figure 5 , Figure 5 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a UAV autonomous maneuvering decision-making method based on spatiotemporal situational differentiation. This method includes: obtaining a first spatial position of a first UAV after executing an autonomous maneuvering decision and a second spatial position of a second UAV, wherein the first UAV and the second UAV are located in a UAV adversarial simulation environment; based on the first and second spatial positions, determining a first shortest distance between the first UAV and the attack zone of the second UAV at each time step, and a second shortest distance between the second UAV and the attack zone of the first UAV; based on the first and second shortest distances, determining the distance between the first UAV and the second UAV... The relative attitude values ​​between the two drones are obtained; the centroid distance between the first and second drones is obtained; the product of the relative attitude values ​​and the correction coefficient, and the difference between the centroid distance between the two drones and the maximum attack range of the first drone are determined; the hyperbolic tangent function of the ratio of the product to the square of the difference is used as the reward feedback score of the first drone; the current reward feedback score of the first drone at the current time step and the historical reward feedback score of the first drone at the previous time step are determined; based on the current reward feedback score and the historical reward feedback score, the spatiotemporal situation differential reward function of the first drone is determined, wherein the spatiotemporal situation differential reward function is used to provide positive or negative feedback for autonomous maneuver decisions.

[0211] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0212] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the UAV autonomous maneuvering decision-making method based on spatiotemporal situational difference provided by the above methods. The method includes: obtaining a first spatial position of a first UAV after performing an autonomous maneuvering decision and a second spatial position of a second UAV, wherein the first UAV and the second UAV are located in a UAV adversarial simulation environment; and determining, based on the first spatial position and the second spatial position, a first shortest distance of the first UAV relative to the attack zone of the second UAV at each time step, and a second shortest distance of the second UAV relative to the attack zone of the first UAV. Distance; Based on the first shortest distance and the second shortest distance, determine the relative attitude values ​​between the first UAV and the second UAV; obtain the centroid distance between the first UAV and the second UAV; determine the product of the relative attitude values ​​and the correction coefficient, and the difference between the centroid distance between the two UAVs and the maximum attack range of the first UAV; use the hyperbolic tangent function of the ratio of the product to the square of the difference as the reward feedback score of the first UAV; determine the current reward feedback score of the first UAV at the current time step, and the historical reward feedback score of the first UAV at the previous time step; based on the current reward feedback score and the historical reward feedback score, determine the spatiotemporal situational differential reward function of the first UAV, wherein the spatiotemporal situational differential reward function is used to provide positive or negative feedback for autonomous maneuver decisions.

[0213] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the above-described method for autonomous maneuvering decision-making based on spatiotemporal situational difference for unmanned aerial vehicles (UAVs). This method includes: acquiring a first spatial position of a first UAV after performing an autonomous maneuvering decision and a second spatial position of a second UAV, wherein the first UAV and the second UAV are located in a UAV adversarial simulation environment; based on the first and second spatial positions, determining a first shortest distance between the first UAV and the attack zone of the second UAV at each time step, and a second shortest distance between the second UAV and the attack zone of the first UAV; based on the first and second spatial positions, determining a first shortest distance between the first UAV and the attack zone of the second UAV at each time step; and determining a second shortest distance between the second UAV and the attack zone of the first UAV; and determining a third shortest distance between the first and second shortest distances. At short distances, the relative attitude values ​​between the first and second UAVs are determined; the distance between their centroids is obtained; the product of the relative attitude values ​​and correction coefficients, and the difference between the distance between their centroids and the maximum attack range of the first UAV are determined; the hyperbolic tangent function of the ratio of the product to the square of the difference is used as the reward feedback score of the first UAV; the current reward feedback score of the first UAV at the current time step is determined, along with the historical reward feedback score of the first UAV at the previous time step; based on the current reward feedback score and the historical reward feedback score, the spatiotemporal situational differential reward function of the first UAV is determined, whereby the spatiotemporal situational differential reward function is used to provide positive or negative feedback for autonomous maneuver decisions.

[0214] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0215] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for autonomous maneuver decision of unmanned aerial vehicle based on spatio-temporal situation difference, characterized in that, The method comprises the following steps: acquiring a first spatial position of a first unmanned aerial vehicle after performing an autonomous maneuver decision and a second spatial position of a second unmanned aerial vehicle, wherein the first unmanned aerial vehicle and the second unmanned aerial vehicle are located in an unmanned aerial vehicle combat simulation environment; based on the first spatial position and the second spatial position, determining a first shortest distance of the first unmanned aerial vehicle to an attack zone distance of the second unmanned aerial vehicle at each time step, and a second shortest distance of the second unmanned aerial vehicle to an attack zone of the first unmanned aerial vehicle; based on the first shortest distance and the second shortest distance, determining a relative attitude value between the first unmanned aerial vehicle and the second unmanned aerial vehicle; acquiring a mutual center-of-mass distance between the first unmanned aerial vehicle and the second unmanned aerial vehicle; determining a product of the relative attitude value and a correction coefficient, and a difference between the mutual center-of-mass distance and a maximum attack distance of the first unmanned aerial vehicle; taking a hyperbolic tangent function of a ratio of the product to a square of the difference as a reward feedback score of the first unmanned aerial vehicle; determining a current reward feedback score of the first unmanned aerial vehicle at a current time step, and a historical reward feedback score of the first unmanned aerial vehicle at a previous time step; based on the current reward feedback score and the historical reward feedback score, determining a space-time situation differential reward function of the first unmanned aerial vehicle, wherein the space-time situation differential reward function is used for positive feedback or negative feedback of the autonomous maneuver decision.

2. The method of claim 1, wherein, The method further comprises the following steps before the step of acquiring the first spatial position of the first unmanned aerial vehicle after performing the autonomous maneuver decision and the second spatial position of the second unmanned aerial vehicle: acquiring a spatial position, a flight speed, a pitch angle and a heading angle of each unmanned aerial vehicle in a ground coordinate system; based on the spatial position, the flight speed, the pitch angle and the heading angle, determining a state space of each unmanned aerial vehicle; based on a tangential overload, a normal relative overload and a speed rotation angle of each unmanned aerial vehicle, determining an action space of the unmanned aerial vehicle. The method further comprises the following steps before the step of acquiring the first spatial position of the first unmanned aerial vehicle after performing the autonomous maneuver decision and the second spatial position of the second unmanned aerial vehicle: acquiring a spatial position, a flight speed, a pitch angle and a heading angle of each unmanned aerial vehicle in a ground coordinate system; based on the spatial position, the flight speed, the pitch angle and the heading angle, determining a state space of each unmanned aerial vehicle; 3. The method of claim 1, wherein, based on a tangential overload, a normal relative overload and a speed rotation angle of each unmanned aerial vehicle, determining an action space of the unmanned aerial vehicle. ​ ​ ​ determine a dynamic equation of each of the UAVs based on the change value of the flight speed over time, the change value of the pitch angle over time, and the change value of the heading angle over time; determine a kinematic equation of each of the UAVs based on the change value of the spatial position over time; construct a UAV confrontation simulation environment based on the state space, the action space, the dynamic equation, the kinematic equation, and a preset terminal condition.

4. The method of claim 3, wherein, After the UAV confrontation simulation environment is constructed based on the state space, the action space, the dynamic equation, the kinematic equation, and the preset terminal condition, the method further includes: obtain a heading angle, a pitch angle, and a roll angle of a first UAV and a second UAV in a three-dimensional space in the UAV confrontation simulation environment; determine a first rotation matrix of the first UAV and a second rotation matrix of the second UAV based on the heading angle, the pitch angle, and the roll angle, respectively; determine a relative coordinate of the second UAV in a first body coordinate system of the first UAV; convert the second rotation matrix into a converted second rotation matrix in the first body coordinate system based on the relative coordinate; determine a state feature code between the first UAV and the second UAV based on the first rotation matrix and the converted second rotation matrix.

5. The method of claim 1, wherein, After the first spatial position of the first UAV after executing the autonomous maneuver decision and the second spatial position of the second UAV are obtained, the method further includes: initialize an execution policy network, an execution value network, and an experience buffer of a deep reinforcement learning decision model; store a sample generated by the autonomous maneuver decision into the experience buffer to obtain sampling experience data; optimize the execution policy network and the execution value network based on the sampling experience data according to a target training step number to obtain current execution network parameters; update target policy network parameters and target value network parameters based on the current execution network parameters to obtain a UAV game decision model.

6. The method of claim 5, wherein, After the UAV game decision model is obtained based on the current execution network parameters, the target policy network parameters, and the target value network parameters, the method further includes: determine a target state feature code of the second UAV relative to the first UAV in a target time step; input the target state feature code into the UAV game decision model to obtain a target autonomous maneuver decision for the first UAV output by the UAV game decision model.

7. An unmanned aerial vehicle autonomous maneuver decision device based on space-time situation difference, characterized in that, includes: a first obtaining module configured to obtain a first spatial position of a first UAV after executing an autonomous maneuver decision and a second spatial position of a second UAV, wherein the first UAV and the second UAV are located in a UAV confrontation simulation environment; a first determining module configured to determine a first shortest distance of the first UAV to an attack area distance of the second UAV and a second shortest distance of the second UAV to an attack area of the first UAV at each time step based on the first spatial position and the second spatial position; and a second determining module configured to determine a first rotation matrix of the first UAV and a second rotation matrix of the second UAV based on a heading angle, a pitch angle, and a roll angle of the first UAV and the second UAV in a three-dimensional space in the UAV confrontation simulation environment. a second determining module, configured to determine a relative attitude value between the first unmanned aerial vehicle and the second unmanned aerial vehicle based on the first shortest distance and the second shortest distance; a second obtaining module, configured to obtain a two-end center-of-mass distance between the first unmanned aerial vehicle and the second unmanned aerial vehicle; a third determining module, configured to determine a product of the relative attitude value and a correction coefficient, and a difference between the two-end center-of-mass distance and a maximum attack distance of the first unmanned aerial vehicle; a scoring module, configured to take a hyperbolic tangent function of a ratio of the product and a square of the difference as a reward feedback score of the first unmanned aerial vehicle; a fourth determining module, configured to determine a current reward feedback score of the first unmanned aerial vehicle at a current time step, and a historical reward feedback score of the first unmanned aerial vehicle at a previous time step; a fifth determining module, configured to determine a space-time situation difference reward function of the first unmanned aerial vehicle based on the current reward feedback score and the historical reward feedback score, wherein the space-time situation difference reward function is used to positively or negatively feed back the autonomous motion decision.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the space-time situation difference based autonomous motion decision method of an unmanned aerial vehicle according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the space-time situation difference based autonomous motion decision method of an unmanned aerial vehicle according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the space-time situation difference based autonomous motion decision method of an unmanned aerial vehicle according to any one of claims 1 to 6.

Images