Unmanned aerial vehicle autonomous maneuver decision-making method and device based on space-time situation difference

Abstract

The invention provides an unmanned aerial vehicle autonomous maneuver decision-making method and device based on space-time situation difference, and the method comprises the steps: obtaining a first space position of a first unmanned aerial vehicle after the execution of an autonomous maneuver decision and a second space position of a second unmanned aerial vehicle; determining a first shortest distance of the first unmanned aerial vehicle for the attack area of the second unmanned aerial vehicle and a second shortest distance of the second unmanned aerial vehicle for the attack area of the first unmanned aerial vehicle under each time step; determining a relative attitude value between the first unmanned aerial vehicle and the second unmanned aerial vehicle based on the first shortest distance and the second shortest distance; determining a current reward feedback score of the first unmanned aerial vehicle in the current time step and a historical reward feedback score of the first unmanned aerial vehicle in the previous time step; and based on the current reward feedback score and the historical reward feedback score, a space-time situation differential reward function of the first unmanned aerial vehicle is determined, and the space-time situation differential reward function is used for performing positive feedback or negative feedback on the autonomous maneuver decision.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned system intelligent decision-making, and in particular to a method and device for autonomous maneuvering decision-making of unmanned aerial vehicles based on spatiotemporal situational difference. Background Art

[0002] UAV maneuvering decision-making is the key link for UAVs to achieve autonomous capabilities. UAV maneuvering decision-making refers to making effective maneuvering action choices quickly based on real-time situation information, and its quality directly reflects the intelligence level of the UAV.

[0003] At present, most UAV maneuver decision-making methods based on deep reinforcement learning assume that the adversarial environment is a two-dimensional space and have relatively strict restrictions on the initial state, and often discretize the action space, resulting in poor generalization and versatility of the trained model.

[0004] This shows that the UAV maneuver decision-making method in related technologies has technical problems that make it difficult to meet the needs of real environments. Summary of the invention

[0005] The present invention provides a method and device for autonomous maneuvering decision-making of unmanned aerial vehicles based on spatiotemporal situation difference, so as to solve the defect that the unmanned aerial vehicle maneuvering decision-making method in the prior art is difficult to meet the needs of the real environment, and realize more accurate reward feedback for the unmanned aerial vehicle.

[0006] The present invention provides a method for autonomous maneuvering decision of a UAV based on spatiotemporal situation difference, comprising the following steps. The first spatial position of the first UAV and the second spatial position of the second UAV after executing the autonomous maneuver decision are obtained, wherein the first UAV and the second UAV are located in a UAV confrontation simulation environment; based on the first spatial position and the second spatial position, the first shortest distance of the first UAV to the attack zone of the second UAV and the second shortest distance of the second UAV to the attack zone of the first UAV are determined at each time step; based on the first shortest distance and the second shortest distance, the relative attitude value between the first UAV and the second UAV is determined; the distance between the center of mass of both parties between the first UAV and the second UAV is obtained; the product of the relative attitude value and the correction coefficient and the difference between the center of mass distance of both parties and the maximum attack distance of the first UAV are determined; the hyperbolic tangent function of the ratio of the product to the square of the difference is used as the reward feedback score of the first UAV; the current reward feedback score of the first UAV at the current time step and the historical reward feedback score of the first UAV at the previous time step are determined; based on the current reward feedback score and the historical reward feedback score, the spatiotemporal situation difference reward function of the first UAV is determined, wherein the spatiotemporal situation difference reward function is used to perform positive feedback or negative feedback on the autonomous maneuver decision.

[0007] According to a method for autonomous maneuvering decision of a UAV based on time-space situation difference provided by the present invention, the method for determining the first shortest distance of the first UAV to the attack zone of the second UAV at each time step based on the first spatial position and the second spatial position includes: determining the first mass point of the first UAV and the second mass point of the second UAV respectively based on the first spatial position and the second spatial position; determining the distance between the centers of mass of both parties between the first mass point and the second mass point; obtaining the first flight speed of the first UAV; determining the angle between the vector of the first flight speed and the vector of the distance between the centers of mass of both parties; when the second mass point is located in the rear area of ​​the first mass point and does not exceed the first area of ​​the normal plane of the first flight speed, the distance between the centers of mass of both parties is used as the first shortest distance between the second UAV and the attack zone of the first UAV; when the second mass point is located in the maximum attack range of the first mass point and is not in the second area of ​​the attack cone of the first UAV, the first shortest distance between the second UAV and the attack zone of the first UAV is determined based on the distance between the centers of mass of both parties, the angle and the maximum attack angle of the first UAV.

[0008] According to a method for autonomous maneuvering decision of unmanned aerial vehicles based on spatiotemporal situational difference provided by the present invention, before obtaining the first spatial position of the first unmanned aerial vehicle after executing the autonomous maneuvering decision and the second spatial position of the second unmanned aerial vehicle, the method also includes: obtaining the spatial position, flight speed, pitch angle and heading angle of each unmanned aerial vehicle in the ground coordinate system; determining the state space of each unmanned aerial vehicle based on the spatial position, the flight speed, the pitch angle and the heading angle; determining the action space of each unmanned aerial vehicle based on the tangential overload, normal overload and speed rotation angle of each unmanned aerial vehicle; determining the dynamic equation of each unmanned aerial vehicle based on the change value of the flight speed over time, the change value of the pitch angle over time and the change value of the heading angle over time; determining the kinematic equation of each unmanned aerial vehicle based on the change value of the spatial position over time; constructing a unmanned aerial vehicle confrontation simulation environment based on the state space, the action space, the dynamic equation, the kinematic equation and the preset final condition.

[0009] According to a method for autonomous maneuvering decision of a UAV based on spatiotemporal situational difference provided by the present invention, after constructing a UAV confrontation simulation environment based on the state space, the action space, the dynamic equation, the kinematic equation and the preset final condition, the method also includes: obtaining the heading angle, pitch angle and roll angle of the first UAV and the second UAV in the three-dimensional space of the UAV confrontation simulation environment; based on the heading angle, the pitch angle and the roll angle, respectively determining the first rotation matrix of the first UAV and the second rotation matrix of the second UAV; determining the relative coordinates of the second UAV in the first body coordinate system of the first UAV; based on the relative coordinates, converting the second rotation matrix into a converted second rotation matrix in the first body coordinate system; based on the first rotation matrix and the converted second rotation matrix, determining the state feature coding between the first UAV and the second UAV.

[0010] According to a method for autonomous maneuvering decision-making of unmanned aerial vehicles based on spatiotemporal situational difference provided by the present invention, after obtaining the first spatial position of the first unmanned aerial vehicle after executing the autonomous maneuvering decision and the second spatial position of the second unmanned aerial vehicle, the method also includes: initializing the execution strategy network, execution value network and experience buffer of the deep reinforcement learning decision model; storing the samples generated by the autonomous maneuvering decision into the experience buffer to obtain sampled experience data; optimizing the execution strategy network and the execution value network based on the sampled experience data according to the target training steps to obtain current execution network parameters; updating the target strategy network parameters and the target value network parameters based on the current execution network parameters to obtain the unmanned aerial vehicle game decision model.

[0011] According to a UAV autonomous maneuver decision method based on spatiotemporal situation difference provided by the present invention, after the target strategy network parameters and the target value network parameters are updated based on the current execution network parameters to obtain the UAV game decision model, the method also includes: determining the target state feature code of the second UAV relative to the first UAV within the target time step; inputting the target state feature code into the UAV game decision model to obtain the target autonomous maneuver decision for the first UAV output by the UAV game decision model.

[0012] The present invention also provides an unmanned aerial vehicle autonomous maneuvering decision-making device based on spatiotemporal situation difference, comprising the following modules: a first acquisition module, used to acquire a first spatial position of a first unmanned aerial vehicle and a second spatial position of a second unmanned aerial vehicle after executing an autonomous maneuvering decision, wherein the first unmanned aerial vehicle and the second unmanned aerial vehicle are located in a unmanned aerial vehicle confrontation simulation environment; a first determination module, used to determine, based on the first spatial position and the second spatial position, a first shortest distance of the first unmanned aerial vehicle to an attack zone of the second unmanned aerial vehicle at each time step, and a second shortest distance of the second unmanned aerial vehicle to an attack zone of the first unmanned aerial vehicle; a second determination module, used to determine, based on the first shortest distance and the second shortest distance, a relative attitude value between the first unmanned aerial vehicle and the second unmanned aerial vehicle; a second acquisition module, used to acquire The center-of-mass distance between the first UAV and the second UAV; a third determination module, used to determine the product of the relative attitude value and the correction coefficient, and the difference between the center-of-mass distance between the two parties and the maximum attack distance of the first UAV; a scoring module, used to use the hyperbolic tangent function of the ratio of the product to the square of the difference as the reward feedback score of the first UAV; a fourth determination module, used to determine the current reward feedback score of the first UAV at the current time step and the historical reward feedback score of the first UAV at the previous time step; a fifth determination module, used to determine the spatiotemporal situation difference reward function of the first UAV based on the current reward feedback score and the historical reward feedback score, wherein the spatiotemporal situation difference reward function is used to provide positive or negative feedback to the autonomous maneuver decision.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an autonomous maneuvering decision method for a UAV based on spatiotemporal situational differentiation as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for autonomous maneuvering decision-making of a UAV based on spatiotemporal situational difference as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned unmanned aerial vehicle autonomous maneuvering decision-making methods based on spatiotemporal situational differences.

[0016] The method and device for autonomous maneuvering decision-making of unmanned aerial vehicles based on time-space situation difference provided by the present invention obtain the first spatial position of the first unmanned aerial vehicle after executing the autonomous maneuvering decision and the second spatial position of the second unmanned aerial vehicle in a confrontation simulation environment, so as to determine the shortest distance between the attack area of ​​the first unmanned aerial vehicle and the attack area of ​​the second unmanned aerial vehicle according to the first spatial position and the second spatial position, and then determine the relative attitude value between the first unmanned aerial vehicle and the second unmanned aerial vehicle, thereby realizing accurate quantification of the time-space situation of the two unmanned aerial vehicle confrontation parties; according to the relative attitude value between the first unmanned aerial vehicle and the second unmanned aerial vehicle and the distance between the centers of mass of both parties, the reward feedback score of the first unmanned aerial vehicle is determined, and according to the historical reward feedback score of the first unmanned aerial vehicle, the time-space situation difference reward function is determined, thereby, introducing the idea of ​​time difference can encourage the unmanned aerial vehicle to always maintain a better spatial situation, or explore a maneuvering strategy to turn disadvantage into advantage under disadvantageous conditions; thereby solving the technical problem that the unmanned aerial vehicle maneuvering decision-making method in the related technology is difficult to meet the needs of the real environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the autonomous maneuvering decision method for unmanned aerial vehicle based on spatiotemporal situation difference provided by the present invention.

[0019] Figure 2 It is a simulation experiment effect diagram provided by the present invention.

[0020] Figure 3 It is an overall flow chart of the autonomous maneuvering decision method for unmanned aerial vehicle based on spatiotemporal situation difference provided by the present invention.

[0021] Figure 4 It is a structural flow chart of the unmanned aerial vehicle autonomous maneuvering decision-making device based on spatiotemporal situation difference provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Compared with manned aircraft, drones have advantages such as higher speed, greater overload capacity, more flexible maneuverability and lower cost. Therefore, drones are widely used in various fields and play an irreplaceable role. With the development of information technology and artificial intelligence technology, the autonomous intelligence of drones is also constantly improving. Drone maneuvering decision-making is a key link for drones to achieve autonomous capabilities. Drone maneuvering decision-making refers to making effective maneuvering action choices quickly based on real-time situation information, and its quality directly reflects the intelligence level of drones.

[0025] At present, there are three main types of UAV maneuver decision-making methods: methods based on game theory, methods based on action planning, and methods based on machine learning. Among them, methods based on game theory include differential games, matrix games, etc. This type of method has clear mathematical form and rigorous reasoning process, but the solution complexity is high and it is only suitable for simple maneuver decision-making problems. Methods based on action planning include expert systems, influence diagrams, etc. This type of method simulates and analyzes the decision logic of pilots, selects a series of actions according to the situation, and evaluates and selects the best one, but its rules are rigid and it is difficult to meet the complex and changing battlefield environment. Methods based on machine learning include genetic algorithms, neural networks, Bayesian networks, reinforcement learning, etc. This type of method fits the mapping relationship between situation information and decision sequences in a data-driven way, continuously optimizes the model structure and parameters, and has better adaptability to complex and changing environments. Among them, deep reinforcement learning methods can collect data and obtain rewards and penalties through the interaction between UAVs and the environment, obtain more reasonable action decisions, and have long-term decision-making planning capabilities, which has become a current research hotspot.

[0026] However, most of the current UAV maneuver decision-making methods based on deep reinforcement learning assume that the adversarial environment is a two-dimensional space and have relatively strict restrictions on the initial state, and often discretize the action space, resulting in poor generalization and versatility of the trained model, making it difficult to meet the needs of the real environment. In addition, the existing reinforcement learning methods have high feature information redundancy, and the reward function does not consider spatiotemporal trend information, resulting in low data utilization and time-consuming training.

[0027] In order to solve the above problems in the prior art, the present invention provides a UAV autonomous maneuver decision method based on spatiotemporal situation difference.

[0028] Firstly, the UAV maneuver decision task is modeled as a Markov decision problem in a continuous state space and continuous action space in a three-dimensional space environment, and the initial state of each game is randomly sampled in the state space to enhance the applicability of the model.

[0029] Secondly, based on the rotation matrix, the present invention designs a state feature that mainly uses opponent interaction information and supplements individual information, which effectively reduces information redundancy and improves the learning efficiency of the intelligent agent.

[0030] In addition, the spatial and temporal situation of the two drone opponents can be accurately quantified by combining expert knowledge, spatial geometry information and situation trend changes, giving the intelligent agent more accurate reward feedback and improving the convergence speed of model training.

[0031] Finally, a historical trajectory modeling method is used to enhance the feature information for identifying different opponent tactics, and then combined with a learning method that adjusts the training focus based on adaptive opponent difficulty, the difficulty of model training is greatly reduced, achieving the expected winning rate while taking into account training efficiency.

[0032] refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for autonomous maneuvering decision-making of a UAV based on spatiotemporal situation difference provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0033] Step 101, obtaining a first spatial position of a first UAV and a second spatial position of a second UAV after executing an autonomous maneuver decision, wherein the first UAV and the second UAV are located in a UAV confrontation simulation environment.

[0034] In drone confrontation maneuvering decisions, one should try to keep one's own weak rear away from the opponent's attack area as much as possible, and at the same time, try to make one's own attack area cover the opponent's weak rear as much as possible.

[0035] In an embodiment of the present invention, the UAV confrontation simulation environment includes a state space, an action space, a UAV kinematic and dynamic model, and a final condition of each UAV among a plurality of UAVs.

[0036] In an embodiment of the present invention, the spatial position information of the first UAV after executing the autonomous maneuvering decision and the spatial position information of the second UAV are calculated and recorded in real time.

[0037] Step 102: Based on the first spatial position and the second spatial position, determine the first shortest distance of the first drone to the attack zone of the second drone and the second shortest distance of the second drone to the attack zone of the first drone at each time step.

[0038] In the embodiment of the present invention, and They represent the shortest distance from the blue drone (i.e. the second drone mentioned above) to the attack area of ​​the red drone (i.e. the first drone mentioned above) and the shortest distance from the red drone to the attack area of ​​the blue drone. and Construct a spatial geometric situation function to reflect the situation information of the red and blue parties (i.e. the first UAV and the second UAV) at each time step (time point).

[0039] Step 103: Determine a relative attitude value between the first UAV and the second UAV based on the first shortest distance and the second shortest distance.

[0040] In the embodiment of the present invention, taking the first UAV (for example, the above-mentioned Red team UAV) as an example, the relationship between the two quantities (the first shortest distance and the second shortest distance) can be used to understand whether the Red team is in an advantageous, disadvantageous or balanced maneuvering posture.

[0041] Among them, the relative attitude value of the first UAV can be obtained by express, The value range is [ , ], Used to indicate the maximum attack distance of the first drone.

[0042] When the relative attitude value of the first UAV When it is greater than 0, the maneuvering posture of the first drone is more advantageous than that of the second drone. The larger the value, the greater the chance that the first UAV will tailgate the second UAV, and vice versa.

[0043] Step 104: Obtain the distance between the center of mass of the first UAV and the second UAV.

[0044] In the embodiment of the present invention, in addition to the relative posture value describing the degree of interaction between the own party and the opponent in terms of maneuvering posture, the size of the center of mass distance between the two parties (represented by D) is also one of the indicators for measuring the degree of interaction between the two parties in space.

[0045] Step 105, determining the product of the relative attitude value and the correction coefficient, and the difference between the center of mass distance of both parties and the maximum attack distance of the first UAV.

[0046] Step 106: The hyperbolic tangent function of the ratio of the product to the square of the difference is used as the reward feedback score of the first drone.

[0047] Therefore, in the embodiment of the present invention, based on the spatial geometric relationship formed between the red drone (the first drone) and the blue drone (the second drone) in the combat, the relative attitude value ( ) and the distance between the two sides’ centroids (D) construct a function indicating the reward feedback score of the first drone at a certain moment (time step) , please refer to the following formula (1): (1)

[0048] in, represents the reward feedback score (function), represents the hyperbolic tangent function (activation function), represents the correction factor, Represents the relative attitude value of the first UAV, represents the distance between the two centroids, Indicates the maximum attack distance of the first drone.

[0049] In an embodiment of the present invention, is the attack distance constant of the drone, is the correction factor. In order to encourage the drone to eliminate the opponent near its attack range, hour, Set it to 10 at other times and 5 at other times. Describes the score of the red drone (first drone) in the confrontation with the blue drone (second drone) at different distances and in different maneuvering states at a certain moment. When the distance D between the two sides is smaller, The larger the value, the greater the effect of the red drone on the blue drone, and vice versa. The larger the value, the The larger it is, the easier it is for the Red UAV to put the Blue UAV within its attack range, and vice versa.

[0050] Step 107, determining the current reward feedback score of the first drone at the current time step and the historical reward feedback score of the first drone at the previous time step.

[0051] Step 108: Determine a spatiotemporal situation difference reward function of the first drone based on the current reward feedback score and the historical reward feedback score, wherein the spatiotemporal situation difference reward function is used to provide positive or negative feedback for the autonomous maneuver decision.

[0052] Reward feedback function It can only describe the real-time static spatial geometric state information of the first drone and the second drone, but not the trend information that changes over time. That is, the reward feedback function (score) has nothing to do with the spatial geometric situation of the two opposing parties at the previous moment or the quality of the maneuvering decision made. In order to encourage the intelligent body to always maintain a better spatial situation, or to explore a maneuvering strategy that turns a disadvantage into an advantage under a disadvantageous situation, the embodiment of the present invention introduces the idea of ​​time difference to construct a time-space situation difference reward function. For details, please refer to the following formula (2): (2)

[0053] in, represents the spatiotemporal situational differential reward function, and They represent the reward scores of the red drone (first drone) under the spatial geometric relationship at time t and time t-1 respectively, is the signal adjustment coefficient, for example, it is set to 0.5 by default in the experiment.

[0054] In the embodiment of the present invention, the first item ( ) reflects the reward of the current agent in the spatial geometric relationship; the second item ( ) reflects the changing trend of the agent’s spatial situation from the previous moment to the current moment in terms of time.

[0055] Through the embodiment of the present invention, the design of the spatiotemporal situation differential reward function is an effective signal feedback amplification mechanism. If the second part is a positive value, it means that the intelligent agent has explored excellent maneuvering decisions, allowing the agent (the first drone) to be in a better spatial geometric situation than the previous moment, so a positive feedback signal is given as encouragement; if the second part is a negative value, it means that the decision effect made by the intelligent agent is worse than the previous moment, so a negative feedback signal is given as punishment.

[0056] Through the above steps of the embodiment of the present invention, the first spatial position of the first UAV after executing the autonomous maneuver decision and the second spatial position of the second UAV are obtained, wherein the first UAV and the second UAV are located in a UAV confrontation simulation environment; based on the first spatial position and the second spatial position, the first shortest distance of the first UAV to the attack zone of the second UAV and the second shortest distance of the second UAV to the attack zone of the first UAV are determined at each time step; based on the first shortest distance and the second shortest distance, the relative attitude value between the first UAV and the second UAV is determined; the distance between the center of mass of the first UAV and the second UAV is obtained; the product of the relative attitude value and the correction coefficient, and the distance between the center of mass of the two parties and the maximum attack value of the first UAV are determined. The invention discloses a method for determining a reward feedback score of the first UAV in accordance with the present invention. The method comprises the following steps: a first UAV is provided with a reward feedback score of a first UAV in accordance with the present invention and a second UAV in accordance with the present invention. The first UAV is provided with a reward feedback score of a first ...

[0057] According to a UAV autonomous maneuver decision method based on spatiotemporal situation difference provided by the present invention, based on a first spatial position and a second spatial position, a first shortest distance of a first UAV from an attack zone of a second UAV at each time step is determined, including: Based on the first spatial position and the second spatial position, respectively determine a first mass point of the first drone and a second mass point of the second drone; Determine the distance between the center of mass of the first mass point and the center of mass of the second mass point; Get the first flight speed of the first UAV; Determine the angle between the vector of the first flight speed and the vector of the distance between the centers of mass of both parties; When the second mass point is located in the rear area of ​​the first mass point and does not exceed the first area of ​​the normal plane of the first flight speed, the distance between the centers of mass of both parties is taken as the shortest distance between the second UAV and the attack area of ​​the first UAV; When the second particle is within the maximum attack range of the first particle and is not in the second area of ​​the attack cone of the first drone, the shortest distance between the second drone and the attack area of ​​the first drone is determined based on the distance between the centers of mass of both parties, the angle between them and the maximum attack angle of the first drone.

[0058] In the embodiment of the present invention, the shortest distance ( ) as an example, the shortest distance from the position of the blue drone (the second drone) to the attack area of ​​the red drone (the first drone) in different situations is modeled. Assume that the red and blue drone particles are and , then the physical distance between the mass centers of the two drones is The speed of the red side is , the maximum attack distance is , its attack range is the spherical sector area on the three-dimensional plane, and the maximum attack angle is . The vector connecting the red and blue points (i.e. the distance between the centers of mass of both parties) The angle between them can be referred to the following formula (3): (3)

[0059] in, The angle between the vector representing the first flight speed and the vector representing the distance between the centers of mass of both parties, Indicates the first flight speed, A vector representing the distance between the centers of mass of both parties.

[0060] The following is a method for determining the shortest distance for the blue team (second drone) to enter the red team (first drone) attack area when the red and blue teams are in different spatial positions.

[0061] like In the area behind r and not exceeding the area passing through point r The area of ​​the normal plane, then the blue square Distance to Red Square The shortest distance to the attack zone is: ,in, Represents the distance between the centroids of both parties.

[0062] like Within the maximum attack range of r but not in the attack zone and not in the area mentioned above, then the blue side Distance to Red Square The shortest distance to the attack zone is: ,in, The distance between the two centroids, The angle between the vector representing the first flight speed and the vector representing the distance between the centers of mass of both parties, Indicates the maximum attack angle of the first drone.

[0063] like If the blue side is outside the maximum attack range of r but not within the extended range of the attack cone and not within the aforementioned area, then Distance to Red Square The shortest distance to the attack zone is: ,in , ,in, represents the distance between the two centroids, The angle between the vector representing the first flight speed and the vector representing the distance between the centers of mass of both parties, Indicates the maximum attack angle of the first drone.

[0064] like Outside the maximum attack range of r but within the extended range of the attack cone, the blue side Distance to Red Square The shortest distance to the attack zone is: ,in, The distance between the two centroids, Indicates the maximum attack distance of the first drone.

[0065] According to the above formula, the embodiment of the present invention can determine the shortest distance for the blue side (the second drone) to enter the attack area of ​​the red side (the first drone) in each time step: , and vice versa, the shortest distance for the red side to enter the blue side's attack area can be calculated Taking the red drone as an example, the relationship between these two quantities can be used to understand whether the red drone is in an advantageous, disadvantageous or balanced maneuvering posture. The relative attitude value between the red drone and the blue drone The value range is [ , ],when >0, the Red UAV's maneuvering posture is more advantageous than that of the Blue UAV. The larger it is, the greater the chance that the red team will chase the blue team, and vice versa.

[0066] According to a method for autonomous maneuvering decision of a UAV based on spatiotemporal situation difference provided by the present invention, before obtaining a first spatial position of a first UAV after executing an autonomous maneuvering decision and a second spatial position of a second UAV, the method further includes: Obtain the spatial position, flight speed, pitch angle and heading angle of each drone in the ground coordinate system; Determine the state space of each drone based on spatial position, flight speed, pitch angle, and heading angle; Determine the action space of each drone based on its tangential overload, normal overload, and speed rotation angle; Determine the dynamic equation of each UAV based on the change of flight speed over time, the change of pitch angle over time, and the change of heading angle over time; Determine the kinematic equations of each drone based on the change in spatial position over time; A drone confrontation simulation environment is constructed based on state space, action space, dynamic equations, kinematic equations and preset final conditions.

[0067] In the embodiment of the present invention, the state space definition of the drone can refer to the following formula (4): (4)

[0068] in, represents the state space of the drone, The three components representing the spatial position of the drone in the ground coordinate system; Indicates the flight speed of the drone. Indicates the pitch angle of the drone, Indicates the heading angle (yaw angle), with the left deviation of the nose as positive.

[0069] The three-dimensional vector of the UAV’s motion space is encoded and used as the control quantity to control the UAV’s motion space. For details, please refer to the following formula (5): (5)

[0070] in, represents the action space of the drone, The tangential overload is the ratio of the thrust and drag force on the drone in the direction of speed to the aircraft's own gravity. is the normal overload, which represents the ratio of the resultant force on the plane perpendicular to the velocity direction of the drone to the aircraft’s own gravity; is the velocity roll angle, which represents the angle between the aircraft symmetry plane and the vertical plane where the aircraft's flight direction line is located. Considering the physical properties of the drone, the range of the three components is limited: the normal overload range is , the tangential overload range is , the range of velocity roll angle is .

[0071] The three-degree-of-freedom dynamic equation is established in the UAV (aircraft) track coordinate system, and the specific reference is the following formula (6): (6)

[0072] in, Indicates the flight speed of the drone. represents the time step, represents gravity, Indicates the tangential overload of the drone. Indicates the pitch angle of the drone, represents the normal overload of the drone, is the velocity roll angle of the drone, Indicates the heading angle of the drone.

[0073] Through numerical integration, we can obtain the values ​​of the aircraft's speed, pitch angle, and yaw angle changing with time after the action is performed. Then, in the ground coordinate system, the aircraft kinematic equation is established. For details, please refer to the following formula (7): (7)

[0074] in, The three components representing the spatial position of the drone in the ground coordinate system, represents the time step, Indicates the flight speed of the drone. Indicates the pitch angle of the drone, Indicates the heading angle of the drone.

[0075] By solving the differential equation, we can get the position of the aircraft in the three-dimensional space in the new state. The two opposing sides use the above dynamics and kinematics equations to calculate the new state information after each action is completed.

[0076] In the embodiment of the present invention, the final conditions of the drone confrontation are set to six situations, namely, red side wins, blue side wins, draw, red side out of bounds, blue side out of bounds and timeout, and the rewards given to the corresponding final states are 300, -100, -0, -100, 0 and 0 respectively. The specific final condition determination method refers to the following formula (8): (8)

[0077] in, is the spatial position vector pointing from the red side (first drone) to the blue side (second drone), is the angle between the speed of the red drone and the distance between the centers of mass of both parties, It is the angle between the speed of the blue drone and the distance between the centers of mass of both parties.

[0078] According to a method for autonomous maneuvering decision of a UAV based on spatiotemporal situation difference provided by the present invention, after constructing a UAV confrontation simulation environment based on state space, action space, dynamic equations, kinematic equations and preset final conditions, the method further includes: Obtaining the heading angle, pitch angle, and roll angle of the first UAV and the second UAV in three-dimensional space in the UAV confrontation simulation environment; Based on the heading angle, the pitch angle, and the roll angle, respectively determine a first rotation matrix of the first UAV and a second rotation matrix of the second UAV; Determine the relative coordinates of the second UAV in the first body coordinate system of the first UAV; Based on the relative coordinates, the second rotation matrix is ​​converted into a converted second rotation matrix in the first body coordinate system; Based on the first rotation matrix and the converted second rotation matrix, a state feature code between the first UAV and the second UAV is determined.

[0079] In three-dimensional space, the attitude angles of the drone are the heading angle , Pitch angle and roll angle , which represent the angles of rotation of the aircraft around the x, y, and z axes respectively. Next, the rotation matrices of the corresponding angles of rotation around each axis are calculated respectively. For details, please refer to the following formula (9): (9)

[0080] Among them, in three-dimensional space ( ) The attitude angles of the drone are respectively the heading angle , Pitch angle and roll angle .

[0081] According to the rotation order of coordinate axes z, y, and x, calculate the rotation matrix from the aircraft body coordinate system to the inertial coordinate system , please refer to the following formula (10): (10)

[0082] in, Indicates the heading angle of the drone, Indicates the pitch angle of the drone, Indicates the roll angle of the drone.

[0083] Substituting the corresponding Euler angle of the red side (the first drone), we get its rotation matrix: Similarly, the rotation matrix of the blue side (the second drone) is .

[0084] Assuming that the red party's coordinates are at the origin, the blue party's coordinates need to be translated, then the blue party's relative coordinates are You can refer to the following formula (11)

[0085] (11)

[0086] in, Represents the relative coordinates of the blue square, Represent the horizontal axis, vertical axis, and vertical axis respectively. represents the blue square (blue), Represents the red square (red), Represents the transposed matrix.

[0087] Combined with the red square's rotation matrix The relative coordinates of the blue side in the red side's body coordinate system can be obtained , please refer to the following formula (12): (12)

[0088] in, Represents the spatial rotation of the three-dimensional space from the inertial coordinate system to the red team's body coordinate system, Represents the relative coordinates of the blue square, Indicates the relative coordinates of the blue team in the red team's body coordinate system. The coordinates in the red team's body coordinate system are expressed as: =( , ,

[0089] The rotation matrix of the blue square Transform the rotation matrix to the Red Team's body coordinate system , please refer to the following formula (13): (13)

[0090] in, represents the rotation matrix of the blue square, Represents the spatial rotation of the three-dimensional space from the inertial coordinate system to the red team's body coordinate system, Indicates the transformation of the blue team's rotation matrix to the red team's body coordinate system.

[0091] make Middle Line The elements of the column are recorded as . The heading angle corresponding to the blue side in the red side coordinate system is solved by inversely solving the rotation matrix , Pitch angle and roll angle , refer to the following formula (14) for details: (14)

[0092] In summary, through the conversion from the inertial coordinate system to the red team's body coordinate system, the following red and blue team state feature coding representation in the UAV confrontation environment is constructed:

[0093] The state representation dimension is 10, which consists of three parts: the first part ( ) is the altitude, velocity and pitch angle of the red team in the absolute coordinate system. Here, the attitude angle of the red team only cares about the pitch angle related to the gravity direction and the flight limit altitude. , own heading angle It is irrelevant to this, so there is no need to consider it; the second part is the relative speed between the blue side and the red side ; The third part is the blue coordinates in the red body coordinate system calculated based on the rotation matrix ( , , ) and the blue side attitude angle ( ).

[0094] In the red team's body coordinate system, the red team's coordinates and attitude angles are both 0. The blue team's new coordinates and new attitude angles are the state information of the opponent perceived by the red team from the first-person perspective, which greatly enhances the situational awareness of the intelligent agent and makes it easier for the decision-making algorithm to learn and deduce the relationship between states, thereby effectively reducing information redundancy and improving the decision-making learning efficiency of the intelligent agent.

[0095] According to a method for autonomous maneuvering decision of a UAV based on spatiotemporal situation difference provided by the present invention, after obtaining a first spatial position of a first UAV after executing an autonomous maneuvering decision and a second spatial position of a second UAV, the method further includes: Initialize the execution strategy network, execution value network, and experience buffer of the deep reinforcement learning decision model; The samples generated by the autonomous maneuvering decision are stored in the experience buffer to obtain sampled experience data; According to the target training steps and based on the sampled experience data, the execution strategy network and the execution value network are optimized to obtain the current execution network parameters; Based on the current execution network parameters, the target strategy network parameters and the target value network parameters are updated to obtain the drone game decision model.

[0096] In the embodiment of the present invention, a Twin Delayed Deep Deterministic policy gradient algorithm (TD3) is used to train the maneuver decision model, initialize the policy network, value network and experience buffer of the deep reinforcement learning decision model, store the samples generated by the UAV maneuver decision process in the buffer and sample the experience data, optimize the value network parameters by gradient descent, optimize the policy network parameters by policy gradient at a certain training step, and update the target policy network and target value network parameters by the current network parameters. For specific training steps, refer to the following steps.

[0097] Step 1: Initialize the parameters of the execution strategy network respectively , parameters of the target policy network , parameters of the two execution value networks and , the parameters of the two target value networks and , experience buffer, action noise , and synchronize the parameters of each target network with the parameters of the corresponding execution network.

[0098] Step 2: Randomly initialize the states of the two adversaries and preprocess them to obtain the overall state of the environment .

[0099] Step 3: Based on the overall state of the environment , using the execution policy network with added noise to select actions , after executing the action, get the next state and rewards .

[0100] Step 4: The sample data is stored in the buffer.

[0101] Step 5: Sample N batches of empirical data from the buffer , the gradient descent method is used to iteratively optimize the parameters of the execution value network according to the loss function.

[0102] The loss function can be specifically referred to the following formula (15): (15)

[0103] in, represents the loss function, Indicates The parameters of the Q network, Indicates the execution strategy network selection action Rewards, represents the discount factor, Indicates the next state after executing the action. Indicates the use of parameters The target policy network gets the next state after executing the action. represents the exploration noise, Indicates the use of parameters The Q network is in the state action Q value prediction under Represents the batch size, that is, the number of samples used in one update.

[0104] Step 6: After a certain number of training steps, use the policy gradient method to optimize the parameters of the execution policy network.

[0105] Specifically, please refer to the following formula (16): (16)

[0106] in, represents the policy gradient, Indicates in status Next, for action Q value About Action The gradient of The gradient is Equal Strategy The output actions are calculated. Representation strategy About parameters The gradient of , which is used to indicate how to change the policy parameters to affect the state Next, select the action. Represents the batch size, that is, the number of samples used in one update.

[0107] Step 7: If the current state is final, continue from step 2; if the current state is not final, As the current state, continue execution from step 3.

[0108] Step 8: When the training goal is reached, save the target network as a model for drone game decision-making.

[0109] According to a method for autonomous maneuvering decision-making of a UAV based on spatiotemporal situation difference provided by the present invention, after updating the target strategy network parameters and the target value network parameters based on the current execution network parameters to obtain the UAV game decision model, the method further includes: Determine a target state feature encoding of the second UAV relative to the first UAV within a target time step; The target state feature code is input into the UAV game decision model to obtain the target autonomous maneuver decision for the first UAV output by the UAV game decision model.

[0110] In the embodiment of the present invention, when the situation of the opponent is uncertain, an adaptive opponent training method is proposed.

[0111] When the adversarial environment switches to a different type of opponent to fight against the red team, since different types of opponents adopt different strategies, the maneuver strategy previously learned by the agent against a certain type of opponent may not be applicable to other types of opponents. If the agent continues to be trained to adapt to the strategy of the current new opponent, it will forget the strategy against the previous opponent or the training will not converge.

[0112] To cope with uncertain opponents, we combined the implicit opponent modeling method with the multi-task learning method and proposed an adaptive opponent training method. First, we enhanced the feature information of the opponent's different tactics based on historical trajectory modeling, then set the reward in the adversarial environment as a reward function based on the temporal and spatial situational difference, and finally, dynamically adjusted the sampling weight of each opponent in the opponent pool during the training process based on the adversarial results, so as to achieve the expected winning rate while taking into account the training efficiency.

[0113] Among them, the above-mentioned feature information of enhancing the recognition of different tactics of the opponent based on historical trajectory modeling can be specifically: based on the state encoding method of converting the absolute coordinate system to the body coordinate system, the opponent type representation learning is performed on the opponent's previous k steps of historical trajectory, and the ability to recognize different types of opponents is improved by learning the opponent's movement trend characteristics. For details, please refer to the following steps: Step 1: Initialize a global queue container que and set the queue size to L to store the opponent's historical state encoding information for the previous L steps.

[0114] Step 2: When the UAV game confrontation simulation platform performs the tth time step, the original state encoding information is obtained , state encoding method based on coordinate system conversion get .

[0115] in, Indicates the state feature encoding after the coordinate system is converted. For details, please refer to the above and will not be repeated here.

[0116] Step 3, determine the blue team's state code.

[0117] The blue team’s state encoding can be specifically referred to in the following formula (17): (17)

[0118] in, Indicates the conversion of the inertial coordinate system of the blue body to the coordinate system of the red body. The blue coordinates in the red body coordinate system ( , , ) and the blue side attitude angle ( ), Indicates the speed change of the blue team's body.

[0119] Step 4: Combine the opponent state encoding information of the first L time steps stored in the queue container to form the final input of the policy network .

[0120] The final input can refer to the following formula (18): = ( (18)

[0121] in, Represents the final input, Indicates the state feature encoding, Indicates the status code of the blue team. represents the time step, Represents a time step queue.

[0122] Step 5: Input to the current policy network to get the action .

[0123] Step 6: Move Input into the simulation platform and get rewarded and the original state encoding information of the next time step .

[0124] Step 7, repeat steps 3 and 4 to process ,get .

[0125] Step 8, finally get the tuple , stored in the experience replay pool.

[0126] in, Indicates status code information, Indicates action, Indicates reward, Indicates the next state encoding information.

[0127] Embed the opponent's historical trajectory information into the state encoding information It can effectively improve the ability of the intelligent agent to distinguish different types of opponents, and then output the optimal maneuvers according to different types of opponents to defeat the opponents.

[0128] In the embodiment of the present invention, the sampling weight of each opponent in the opponent pool is dynamically adjusted during the training process based on the confrontation results.

[0129] Specifically, in order to improve the ability of the intelligent agent (drone) to adapt to different opponent strategies, that is, to adapt to different tasks, a method is proposed that can dynamically switch the game tasks being trained during the training process. Specifically, the training framework is adjusted, and the results of the confrontation with each opponent are recorded during the training phase. Before each game starts, the opponent pool is evenly sampled as the opponent of the game. As the training progresses, the sampling weight of each opponent in the opponent pool is automatically adjusted based on the confrontation results. More difficult opponents get larger weights, and relatively easy opponents get smaller weights.

[0130] Using this training method that automatically adjusts the opponent's appearance ratio based on the confrontation results can achieve the effect of accelerating the learning convergence of the decision network. The specific implementation steps are as follows: Step 1: Initialize N queues Q of size L to record the historical game results of N opponents.

[0131] Step 2: Randomly initialize a relative weight list W with a capacity of N.

[0132] Step 3: Before each game starts, randomly select an opponent based on W.

[0133] Step 4: After each game is over, the confrontation results are updated to the game results queue of the corresponding opponent. middle.

[0134] Step 5: After every M games, update W by counting the non-winning rate (opponent wins, out of bounds, timeout) of the game against each opponent.

[0135] Step 6: Repeat steps 3 to 5 until the training is completed.

[0136] In summary, the present invention models the UAV maneuver decision task as a Markov decision problem of continuous state space and continuous action space in a three-dimensional space environment, and constructs the state feature coding representation of the red and blue sides by transforming the absolute coordinate system and the body coordinate system. It mainly uses the opponent's interaction information and the individual information as a supplement, which effectively reduces the redundancy of state information and improves the learning efficiency of intelligent agent samples. In addition, the initial state of each game is randomly sampled in the state space, which enhances the versatility of the model.

[0137] The present invention combines expert knowledge with spatial geometric information to quantify the spatial situation of the two opposing drones, and introduces the idea of ​​temporal difference to characterize the trend change of the situation, thereby constructing a temporal and spatial situation differential reward function, giving the drone more accurate reward feedback, overcoming the reward sparsity problem, and improving the model convergence speed and decision-making ability.

[0138] In order to cope with the situation of uncertain opponents, the present invention combines implicit opponent modeling with multi-task learning and proposes an adaptive opponent training method, which uses historical trajectory modeling to enhance the feature information of identifying different tactics of opponents, and dynamically adjusts the sampling weight of each opponent in the opponent pool based on the confrontation results during the training process, which greatly reduces the difficulty of model training and takes into account the training efficiency while achieving the expected winning rate.

[0139] The following describes a specific implementation method of a UAV autonomous maneuver decision-making method based on spatiotemporal situation difference provided by the present invention in practical applications.

[0140] refer to Figure 2 , Figure 2 It is a simulation experiment effect diagram provided by the present invention.

[0141] In the simulated confrontation environment, the normal overload range of the action space is set to , the tangential overload range is , the range of velocity roll angle is The attack distance is set to 400 meters, the gravity acceleration is set to 9.8, and the time step is set to 1, that is, the execution time of each action is 1 second. The coordinates, speeds, and Euler angles of both sides in each round of the training environment are initialized by uniform random sampling. The initialization range of the states of both sides is shown in Table 1.

[0142] Table 1: Status initialization range of both parties

[0143] The parameters of the reinforcement learning algorithm TD3 used are as follows: the number of training starting steps is 10,000, the experience replay pool size is 100,000, the batch size is 100, the learning rate of the actor network and the evaluator network is 5e-4, the Gaussian noise variance of the enhanced exploration is 0.1, and the discount rate is 0.99. We use the red team's winning rate as the evaluation indicator, count the number of games in which the red team wins in the last 1,000 games, and then divide it by 1,000 to convert it into a percentage.

[0144] The present invention conducts experimental verification under the condition of uncertain opponents. The opponent pool contains two types of strategies: steady-state linear strategy and Min-Max strategy. The state coding of coordinate system conversion and spatiotemporal situation differential reward function proposed in the present invention are used to compare the results under different training methods, including three training methods: opponent uniform sampling, opponent trajectory modeling + uniform sampling, and opponent trajectory modeling + adaptive sampling.

[0145] Table 2 shows the winning rate of the red side against the blue side with different strategies under different training methods, where the maximum number of training steps is set to 2.5 million steps. It can be seen that the training method of uniform sampling of opponents has not converged after reaching the maximum number of training steps, and its winning rate against the Min-Max strategy is still relatively low; the opponent trajectory modeling + uniform sampling training method achieves the result that the red side model can simultaneously defeat all opponents in the opponent pool with a winning rate of not less than 0.8 after 2.5 million steps; the opponent trajectory modeling + adaptive sampling training method converges at 1.5 million steps, and achieves a winning rate of not less than 0.84 for defeating all opponents in the opponent pool at the same time. Therefore, the method proposed by the present invention can improve the sample utilization rate, increase the model convergence speed and improve the accuracy of the model's strategy generation.

[0146] Table 2: The winning rate of the red team against the blue team with different strategies under different training methods

[0147] refer to Figure 3, Figure 3 This is an overall flow chart of the UAV autonomous maneuver decision-making method based on spatiotemporal situation difference provided by the present invention. For details, please refer to the following steps.

[0148] Step 1: Model the UAV maneuver decision task as a Markov decision problem with continuous state space and continuous action space in a three-dimensional space environment, establish an adversarial simulation environment, including UAV kinematics and dynamics modeling, and define the state space, action space, and final conditions.

[0149] Step 2: Randomly initialize the state information of both parties in each game in the state space to enhance the practicality of the model, design a state encoding method based on coordinate system transformation, and use state features based on opponent interaction information as the main and individual information as the auxiliary, effectively reducing information redundancy and improving the learning efficiency of the intelligent agent.

[0150] Step 3: Design a reward function based on spatiotemporal situational differences, which combines expert knowledge, spatial geometric information, and situation trend changes to accurately quantify the spatiotemporal situation of both drone opponents, giving the intelligent agent more accurate reward feedback and improving the convergence speed of model training.

[0151] Step 4, train the maneuver decision model: initialize the policy network, value network and experience buffer of the deep reinforcement learning decision model, store the samples generated by the UAV maneuver decision process into the buffer and sample the experience data, use gradient descent to optimize the value network parameters, and use the policy gradient to optimize the policy network parameters every certain number of training steps, and use the current network parameters to update the target policy network and target value network parameters.

[0152] Step 5: For the case of uncertain opponents, an adaptive opponent training method is proposed. The feature information of different tactics of the opponent is enhanced based on historical trajectory modeling. Then, the sampling weight of each opponent in the opponent pool is dynamically adjusted during the training process based on the confrontation results, thereby achieving the expected winning rate while taking into account the training efficiency.

[0153] Step 6, repeat steps 3-5 until the final state is reached, and use the target network as a model for the UAV's maneuver decision.

[0154] The following is a description of the UAV autonomous maneuvering decision-making device based on spatiotemporal situation difference provided by the present invention. The UAV autonomous maneuvering decision-making device based on spatiotemporal situation difference described below and the UAV autonomous maneuvering decision-making method based on spatiotemporal situation difference described above can be referenced to each other.

[0155] refer to Figure 4 , Figure 4It is a structural flow chart of the UAV autonomous maneuver decision-making device based on spatiotemporal situation difference provided by the present invention, which includes: a first acquisition module 401, a first determination module 402, a second determination module 403, a second acquisition module 404, a third determination module 405, a scoring module 406, a fourth determination module 407 and a fifth determination module 408.

[0156] A first acquisition module 401 is used to acquire a first spatial position of a first UAV and a second spatial position of a second UAV after the first UAV performs an autonomous maneuver decision, wherein the first UAV and the second UAV are located in a UAV confrontation simulation environment; A first determination module 402 is used to determine, based on the first spatial position and the second spatial position, a first shortest distance of the first drone from the attack zone of the second drone and a second shortest distance of the second drone from the attack zone of the first drone at each time step; A second determination module 403 is used to determine a relative attitude value between the first UAV and the second UAV based on the first shortest distance and the second shortest distance; The second acquisition module 404 is used to acquire the distance between the center of mass of the first UAV and the second UAV; The third determination module 405 is used to determine the product of the relative attitude value and the correction coefficient, and the difference between the center of mass distance of both parties and the maximum attack distance of the first UAV; A scoring module 406 is configured to use a hyperbolic tangent function of a ratio of a product to a square of a difference as a reward feedback score of the first drone; The fourth determination module 407 is used to determine the current reward feedback score of the first drone at the current time step and the historical reward feedback score of the first drone at the previous time step; The fifth determination module 408 is used to determine the spatiotemporal situation difference reward function of the first drone based on the current reward feedback score and the historical reward feedback score, wherein the spatiotemporal situation difference reward function is used to provide positive feedback or negative feedback for the autonomous maneuver decision.

[0157] Specifically, the above-mentioned UAV autonomous maneuvering decision-making device based on spatiotemporal situation difference provided by the present invention can implement all the method steps implemented by the above-mentioned UAV autonomous maneuvering decision-making method embodiment based on spatiotemporal situation difference, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0158] refer to Figure 5 , Figure 5 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communications interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the autonomous maneuvering decision method of the UAV based on the time-space situation difference, the method comprising: obtaining the first spatial position of the first UAV after executing the autonomous maneuvering decision and the second spatial position of the second UAV, wherein the first UAV and the second UAV are located in the UAV confrontation simulation environment; based on the first spatial position and the second spatial position, determining the first shortest distance of the first UAV to the attack zone of the second UAV at each time step, and the second shortest distance of the second UAV to the attack zone of the first UAV; based on the first shortest distance and the second shortest distance, determining the distance between the first UAV and the second UAV The method comprises the steps of: determining a relative attitude value between the first UAV and the second UAV; obtaining a center of mass distance between the first UAV and the second UAV; determining the product of the relative attitude value and the correction coefficient, and the difference between the center of mass distance between the two parties and the maximum attack distance of the first UAV; taking the hyperbolic tangent function of the ratio of the product to the square of the difference as the reward feedback score of the first UAV; determining the current reward feedback score of the first UAV at the current time step and the historical reward feedback score of the first UAV at the previous time step; determining a spatiotemporal situation difference reward function of the first UAV based on the current reward feedback score and the historical reward feedback score, wherein the spatiotemporal situation difference reward function is used to provide positive or negative feedback to the autonomous maneuver decision.

[0159] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0160] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the UAV autonomous maneuver decision-making method based on spatiotemporal situation difference provided by the above methods, and the method includes: obtaining a first spatial position of a first UAV and a second spatial position of a second UAV after executing an autonomous maneuver decision, wherein the first UAV and the second UAV are located in a UAV confrontation simulation environment; based on the first spatial position and the second spatial position, determining the first shortest distance of the first UAV to the attack zone of the second UAV at each time step, and the second shortest distance of the second UAV to the attack zone of the first UAV. distance; based on the first shortest distance and the second shortest distance, determine the relative attitude value between the first UAV and the second UAV; obtain the center of mass distance between the first UAV and the second UAV; determine the product of the relative attitude value and the correction coefficient, and the difference between the center of mass distance between the two parties and the maximum attack distance of the first UAV; use the hyperbolic tangent function of the ratio of the product to the square of the difference as the reward feedback score of the first UAV; determine the current reward feedback score of the first UAV at the current time step and the historical reward feedback score of the first UAV at the previous time step; based on the current reward feedback score and the historical reward feedback score, determine the spatiotemporal situation difference reward function of the first UAV, wherein the spatiotemporal situation difference reward function is used to provide positive or negative feedback for autonomous maneuvering decisions.

[0161] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the UAV autonomous maneuvering decision method based on spatiotemporal situation difference provided by the above methods, the method comprising: obtaining a first spatial position of a first UAV and a second spatial position of a second UAV after executing an autonomous maneuvering decision, wherein the first UAV and the second UAV are located in a UAV confrontation simulation environment; based on the first spatial position and the second spatial position, determining a first shortest distance of the first UAV to the attack zone of the second UAV at each time step, and a second shortest distance of the second UAV to the attack zone of the first UAV; based on the first shortest distance and the second shortest distance Short distance, determine the relative attitude value between the first UAV and the second UAV; obtain the center of mass distance between the first UAV and the second UAV; determine the product of the relative attitude value and the correction coefficient, and the difference between the center of mass distance between the two parties and the maximum attack distance of the first UAV; use the hyperbolic tangent function of the ratio of the product to the square of the difference as the reward feedback score of the first UAV; determine the current reward feedback score of the first UAV at the current time step and the historical reward feedback score of the first UAV at the previous time step; based on the current reward feedback score and the historical reward feedback score, determine the spatiotemporal situation difference reward function of the first UAV, wherein the spatiotemporal situation difference reward function is used to provide positive or negative feedback for autonomous maneuvering decisions.

[0162] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for autonomous maneuvering decision of unmanned aerial vehicles based on spatiotemporal situation difference, characterized in that: include: Acquire a first spatial position of a first UAV and a second spatial position of a second UAV after the first UAV executes an autonomous maneuver decision, wherein the first UAV and the second UAV are located in a UAV confrontation simulation environment; Based on the first spatial position and the second spatial position, determining a first shortest distance of the first UAV from the attack zone of the second UAV and a second shortest distance of the second UAV from the attack zone of the first UAV at each time step; Determining a relative attitude value between the first UAV and the second UAV based on the first shortest distance and the second shortest distance; Obtaining a distance between the center of mass of the first UAV and the second UAV; Determine the product of the relative attitude value and the correction coefficient, and the difference between the center-of-mass distance between the two parties and the maximum attack distance of the first UAV; Using a hyperbolic tangent function of the ratio of the product to the square of the difference as a reward feedback score for the first drone; Determine a current reward feedback score of the first drone at a current time step and a historical reward feedback score of the first drone at a previous time step; Based on the current reward feedback score and the historical reward feedback score, a spatiotemporal situation difference reward function of the first UAV is determined, wherein the spatiotemporal situation difference reward function is used to provide positive feedback or negative feedback to the autonomous maneuver decision.

2. The autonomous maneuvering decision method for unmanned aerial vehicles based on spatiotemporal situational difference according to claim 1 is characterized in that: The determining, based on the first spatial position and the second spatial position, a first shortest distance of the first UAV from the attack zone of the second UAV at each time step includes: Based on the first spatial position and the second spatial position, respectively determine a first mass point of the first UAV and a second mass point of the second UAV; Determining a distance between the centroids of the first mass point and the second mass point; Obtaining a first flight speed of the first UAV; Determine the angle between the vector of the first flight speed and the vector of the distance between the centers of mass of both parties; When the second mass point is located in the rear area of ​​the first mass point and does not exceed the first area of ​​the normal plane of the first flight speed, the distance between the mass centers of both parties is taken as the first shortest distance between the second UAV and the attack area of ​​the first UAV; When the second particle is within the maximum attack range of the first particle and is not in the second area of ​​the attack cone of the first UAV, the first shortest distance between the second UAV and the attack area of ​​the first UAV is determined based on the distance between the centers of mass of both parties, the angle and the maximum attack angle of the first UAV.

3. The autonomous maneuvering decision-making method for unmanned aerial vehicles based on spatiotemporal situational difference according to claim 1 is characterized in that: Before obtaining the first spatial position of the first UAV after executing the autonomous maneuvering decision and the second spatial position of the second UAV, the method further includes: Obtain the spatial position, flight speed, pitch angle and heading angle of each drone in the ground coordinate system; Determine a state space of each drone based on the spatial position, the flight speed, the pitch angle, and the heading angle; Determine the action space of each UAV based on the tangential overload, normal overload and speed rotation angle of each UAV; Determine a dynamic equation of each drone based on the change in the flight speed over time, the change in the pitch angle over time, and the change in the heading angle over time; Determining the kinematic equation of each drone based on the change in the spatial position over time; Based on the state space, the action space, the dynamic equation, the kinematic equation and the preset final condition, a drone confrontation simulation environment is constructed.

4. The autonomous maneuvering decision method for unmanned aerial vehicles based on spatiotemporal situational difference according to claim 3 is characterized in that: After constructing the drone confrontation simulation environment based on the state space, the action space, the dynamic equation, the kinematic equation and the preset final condition, the method further includes: Obtaining the heading angle, pitch angle, and roll angle of the first UAV and the second UAV in the three-dimensional space of the UAV confrontation simulation environment; Based on the heading angle, the pitch angle, and the roll angle, respectively determine a first rotation matrix of the first UAV and a second rotation matrix of the second UAV; Determine the relative coordinates of the second UAV in the first body coordinate system of the first UAV; Based on the relative coordinates, converting the second rotation matrix into a converted second rotation matrix in the first body coordinate system; Based on the first rotation matrix and the converted second rotation matrix, a state feature code between the first UAV and the second UAV is determined.

5. The autonomous maneuvering decision-making method for unmanned aerial vehicles based on spatiotemporal situational difference according to claim 1 is characterized in that: After obtaining the first spatial position of the first UAV after executing the autonomous maneuvering decision and the second spatial position of the second UAV, the method further includes: Initialize the execution strategy network, execution value network, and experience buffer of the deep reinforcement learning decision model; storing samples generated by the autonomous maneuvering decision into the experience buffer to obtain sampled experience data; Optimizing the execution strategy network and the execution value network based on the sampled experience data according to the target training steps to obtain current execution network parameters; Based on the current execution network parameters, the target strategy network parameters and the target value network parameters are updated to obtain the drone game decision model.

6. The autonomous maneuvering decision method for unmanned aerial vehicles based on spatiotemporal situational difference according to claim 5 is characterized in that: After the target strategy network parameters and the target value network parameters are updated based on the current execution network parameters to obtain the drone game decision model, the method further includes: Determine a target state feature encoding of the second UAV relative to the first UAV within a target time step; The target state feature code is input into the UAV game decision model to obtain the target autonomous maneuver decision for the first UAV output by the UAV game decision model.

7. An autonomous maneuvering decision-making device for unmanned aerial vehicles based on spatiotemporal situational difference, characterized in that: include: A first acquisition module is used to acquire a first spatial position of a first UAV and a second spatial position of a second UAV after the first UAV performs an autonomous maneuver decision, wherein the first UAV and the second UAV are located in a UAV confrontation simulation environment; A first determination module is used to determine, based on the first spatial position and the second spatial position, a first shortest distance of the first drone from the attack zone of the second drone and a second shortest distance of the second drone from the attack zone of the first drone at each time step; A second determining module, configured to determine a relative attitude value between the first UAV and the second UAV based on the first shortest distance and the second shortest distance; A second acquisition module, configured to acquire a distance between the center of mass of the first UAV and the second UAV; A third determination module is used to determine the product of the relative attitude value and the correction coefficient, and the difference between the center-of-mass distance between the two parties and the maximum attack distance of the first UAV; a scoring module, configured to use a hyperbolic tangent function of a ratio of the product to the square of the difference as a reward feedback score of the first drone; a fourth determination module, configured to determine a current reward feedback score of the first drone at a current time step and a historical reward feedback score of the first drone at a previous time step; The fifth determination module is used to determine the spatiotemporal situation difference reward function of the first UAV based on the current reward feedback score and the historical reward feedback score, wherein the spatiotemporal situation difference reward function is used to provide positive feedback or negative feedback to the autonomous maneuver decision.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the unmanned aerial vehicle autonomous maneuver decision method based on spatiotemporal situational difference as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for autonomous maneuvering decision of a UAV based on spatiotemporal situational difference as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for autonomous maneuvering decision of a UAV based on spatiotemporal situational difference as described in any one of claims 1 to 6 is implemented.

Images