An unmanned aerial vehicle patrol path decision-making method in a complex environment

Abstract

The application discloses a kind of complex environment under unmanned plane patrol path decision method, it is related to unmanned plane path planning technical field, method includes: with each to be patrolled position as node to construct undirected topological graph;According to the topological constraint of undirected topological graph and node importance generates the steady state distribution of each node patrolled by unmanned plane;Generate multiple candidate state transition matrix with steady state distribution;Dynamic environment evaluation is carried out to the obstacle of each candidate state transition matrix corresponding path, to determine the selectable speed and direction of unmanned plane on selected path;According to the weight of decision network dynamic adjustment according to dynamic environment evaluation result, and then generate execution motion command using the decision network after weight adjustment;According to motion command, unmanned plane is controlled and moves between each to be patrolled position.The application can realize the intelligent decision of unmanned plane under complex environment efficiently by dynamic environment evaluation, planning path, adjusting the weight of decision network, generating motion command.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle path planning, and particularly relates to a method for making a patrol path decision of an unmanned aerial vehicle in a complex environment. BACKGROUND

[0002] The existing technology generates a state transition matrix of a Markov chain by using a Metropolis-Hastings algorithm. A global path planning is generated by using an A* algorithm, a local path planning is realized by using a traditional dynamic window method (DWA), motor control commands of translation and rotation speeds are generated, and a collision-free trajectory is ensured in a static environment. A navigation strategy based on a neural network inputs RGB data, a depth image and laser data, trains a network by using reinforcement learning (RL), and directly outputs control commands of the unmanned aerial vehicle based on a current observation.

[0003] Firstly, the existing technology is a scattered state, and there is no connection between independent systems. Secondly, the state transition matrix of the Markov chain generated by the existing technology has a high self-loop characteristic, and is easy to be learned and predicted by a potential attacker (attacking the unmanned aerial vehicle) to patrol a path. Thirdly, the performance of the traditional dynamic window method (DWA) strongly depends on manual adjustment of weight parameters, and different weight configurations are required in different environments, and dynamic obstacles and changing environments cannot be effectively handled. Finally, the deep learning method is based on a current observation to make a prediction, but when facing an unforeseen input, it cannot guarantee that the generated command is safe (may cause a collision). SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a method for making a patrol path decision of an unmanned aerial vehicle in a complex environment, so as to realize efficient intelligent decision-making of the unmanned aerial vehicle in the complex environment.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application provides a method for making a patrol path decision of an unmanned aerial vehicle in a complex environment, and the method comprises the following steps:

[0006] A node is constructed by taking each to-be-patrolled position as a node to construct an undirected topological graph;

[0007] A steady-state distribution of the unmanned aerial vehicle in each node is generated according to a topological constraint and a node importance of the undirected topological graph;

[0008] A plurality of candidate state transition matrices with the steady-state distribution are generated; wherein each candidate state transition matrix comprises a transition probability of the unmanned aerial vehicle between each node;

[0009] Obstacles on a selected path corresponding to each candidate state transition matrix are dynamically environmentally evaluated, so as to determine a selectable speed and direction of the unmanned aerial vehicle on the selected path;

[0010] According to the dynamic environment evaluation result, the weight of the decision network is dynamically adjusted, and then the execution motion command is generated by using the decision network after the weight adjustment.

[0011] According to the execution motion command, the UAV is controlled to move between each of the to-be-patrolled positions.

[0012] In some embodiments, the generation of the steady-state distribution of the UAV patrolling each of the nodes according to the topological constraints and the node importance of the undirected topological graph comprises the following steps:

[0013] The steady-state distribution is generated according to the topological constraints of the undirected topological graph to improve the probability of the UAV transferring to a target node; therefore, a high-value target node and a node with a short attack time need to obtain more patrolling resources, and the probability of the patrolling UAV accessing these nodes is higher.

[0014] The expression of the steady-state distribution is as follows:

[0015]

[0016] wherein, v i ∈ V, v i represents the i-th node, V represents a set of each of the nodes, w i represents the value of the target node v i , a i represents the attack time, μ i represents the importance weight parameter of the node v i , μ j represents the importance weight parameter of the node v j , n is the total number of nodes, represents the steady-state distribution of the target node v i , and the target steady-state distribution represents the probability distribution of the patroller at each target node after a long-term operation of the Markov chain.

[0017] In some embodiments, the generation of the plurality of candidate state transition matrices with the steady-state distribution comprises the following steps:

[0018] The plurality of candidate state transition matrices with the steady-state distribution and the target network topological structure are generated according to a multivariate random adaptive Markov steady-state matrix algorithm, comprising the following steps:

[0019] First stage: hierarchical random matrix generation;

[0020] A first-stage random matrix satisfying the topological constraints of the undirected topological graph is generated, wherein four distribution intervals are used to allocate the self-loop probability p ii , and the remaining non-self-loop probability p ij is generated again.

[0021] Second stage: relative error iterative adjustment algorithm;

[0022] Adjusting the first stage random matrix according to the error between the probability distribution of the first stage random matrix and the steady state distribution, specifically obtaining a second stage random matrix through relative error calculation, error proportion adjustment, row normalization and random periodic disturbance;

[0023] Third stage: similarity value elimination algorithm;

[0024] Eliminating the matrix in the second stage random matrix whose similarity value is greater than a preset similarity threshold value, specifically ensuring that there is no too close value in the matrix through similarity value detection, random adjustment and row normalization, enhancing randomness and creating a more natural random distribution to obtain a third stage random matrix;

[0025] Fourth stage: matrix diversity guarantee algorithm;

[0026] Adjusting the third stage random matrix according to the average absolute error between the probability distributions of the third stage multiple random matrices, specifically ensuring that the generated multiple Markov state transition matrices P k Have enough diversity, not similar matrix variants, finally obtain a fourth stage random matrix as the candidate state transition matrix.

[0027] In some embodiments, the dynamic environment assessment of the obstacles of the path corresponding to each of the candidate state transition matrices includes the following steps:

[0028] The dynamic environment assessment of the obstacles of the path corresponding to each of the candidate state transition matrices is performed according to a dynamic environment adaptability function, the risk weighted sum of each obstacle is summed, and the environment adaptability is quantified;

[0029] The expression of the dynamic environment adaptability function (DEAF) is:

[0030]

[0031] Wherein, e(v, ω): dynamic environment adaptability function, representing the environment adaptability evaluation value of the unmanned aerial vehicle under the given speed command;

[0032] v: linear speed command of the unmanned aerial vehicle, controlling the speed size of the unmanned aerial vehicle advancing or retreating;

[0033] ω: angular velocity command of the unmanned aerial vehicle, controlling the speed size and direction of the unmanned aerial vehicle turning;

[0034] The current system state estimated using weighted observation data, containing the environment representation of obstacle positions and velocities, where z i is the observation data of the i-th time instance of the UAV, and observation weight is calculated by the following expression:

[0035]

[0036] observation data z i weight, quantifying the importance of the observation in state estimation;

[0037] w a , w r , w d are memory weight coefficients for autocorrelation, temporal proximity, and bias, starting from initial empirical values, set according to environmental properties, updated periodically using online adaptive methods;

[0038] A s (z i , Θ): autocorrelation score of the observation, reflecting the temporal correlation of environmental dynamics;

[0039] Θ: parameter set controlling the autocorrelation score calculation;

[0040] R s (z i ): temporal proximity score of the observation, emphasizing the importance of recent data;

[0041] D s (z i , ψ): bias score of the observation, capturing abnormal changes in the environment;

[0042] ψ: bias sensitivity parameter, controlling the algorithm's emphasis on bias;

[0043] M: current number of dynamic obstacles perceived, including other UAVs;

[0044] Obstacle i existence probability calculated based on state estimation Ω, with a value range of [0, 1], with a larger value indicating a higher collision risk, measuring the probability estimate of obstacle i intersecting with the UAV path when the UAV executes speed command (v, ω);

[0045] Relative speed vector between the UAV and obstacle i calculated based on the UAV speed command (v, ω) and obstacle speed information in state estimation Ω, measuring the speed and direction of relative motion between the UAV and the obstacle;

[0046] d i(v, ω, Ω): the predicted minimum distance between the UAV with velocity (v, ω) and the obstacle i, the smaller the value, the higher the potential collision risk;

[0047] K: a normalization constant to ensure the range of the score item is appropriate, determined based on environmental characteristics and UAV dynamics parameters experiment;

[0048] ε: a positive number, a safety parameter to prevent division by zero.

[0049] In some embodiments, the step of dynamically adjusting the weight of the decision network according to the dynamic environment evaluation result comprises the following steps:

[0050] The step of dynamically adjusting the weight of the decision network according to the path corresponding to the target state transition matrix comprises the following steps:

[0051] Design the system state, output action and reward function required by the adjustment decision network Actor-Critic;

[0052] Add a dynamic environment adaptive function to the traditional DWA to obtain a time-varying environment adaptive dynamic window method;

[0053] Combine the decision network Actor-Critic and the time-varying environment adaptive dynamic window method to obtain an AC-time-varying environment adaptive dynamic window method, use the state value estimation V(s) output by the Critic network in the decision network to adjust the weight of the Actor network in the decision network, that is, the weight parameter in the time-varying environment adaptive dynamic window method;

[0054] The Actor-Critic method replaces the recurrent neural network in the traditional Actor-Critic method with a MoEMamba network, which is a network that combines and mixes expert networks and Mamba networks, alternating between the two networks, one layer of mixed expert networks and one layer of Mamba networks, and so on, a total of k layers;

[0055] System state s t All the information contained in the state of the UAV at time t is described, which is the basis for the AC-TEADWA method and provides complete information for the neural network to predict the AC-TEADWA weight, the expression of the system state is:

[0056]

[0057] Where, s t : the system state at time t, which contains the complete state information of the UAV at the t time;

[0058] Distance measurement of UAV to obstacle at current time t in current UAV coordinate system;

[0059] Distance measurement before k control periods in current UAV coordinate system;

[0060] Distance measurement before l control periods in current UAV coordinate system;

[0061] Distance measurement before g control periods in current UAV coordinate system;

[0062] d t : Distance of UAV current position to target;

[0063] φ t : Angle of UAV current position to target;

[0064] ν t : Current translational velocity of UAV;

[0065] ω t : Current rotational velocity of UAV;

[0066] Obstacle distance change between continuous time steps, capturing dynamic information;

[0067] Time derivative of UAV to target distance, indicating whether approaching target;

[0068] UAV current acceleration state vector is linear acceleration of UAV at current time t, is angular acceleration of UAV at current time t;

[0069] ξ t : Environmental complexity index, calculated by entropy value of obstacle distribution;

[0070] Specific form of distance measurement:

[0071] τ represents past time steps, refers to k, l, g in the formula;

[0072] Superscript t-τ represents measurement time;

[0073] Subscript t represents coordinate system of current UAV position;

[0074] N represents measurement point number of laser radar, is distance between UAV position and nearest obstacle in set direction;

[0075] inputting the system state into the decision network, obtaining an action output by the decision network, the action being expressed as:

[0076] a t = (a t , b t , g t , d t , l t );

[0077] wherein a t : heading term weight;

[0078] b t : gap term weight;

[0079] g t : speed term weight;

[0080] d t : curvature distance component, additional weight of the newly extended cost function;

[0081] l t : dynamic environment adaptability function component, additional weight of the newly extended cost function;

[0082] The UAV receives observations or states of the environment and outputs an action a, which is to set the cost function weights for the next control period, which will be used to generate the motion commands of the UAV.

[0083] Design of the reward function:

[0084] The UAV successfully reaches the target position and is given a large positive reward R1; the UAV stays in the same position for multiple control periods, similar to the UAV being stuck, and is given a negative reward R2; the UAV collides with obstacles / or other UAVs / birds during movement, and is given a large penalty, negative R3; the UAV is collided with by a moving obstacle or other UAV when stationary, and is given a negative reward R4; through the design of such a reward function, the reinforcement learning algorithm is guided to learn a safe and effective navigation strategy.

[0085] In some embodiments, the step of implementing the time-varying environment adaptive dynamic window method comprises the following steps:

[0086] The dynamic environment adaptability function is added to the traditional DWA to obtain the time-varying environment adaptive dynamic window method;

[0087] The objective function of the time-varying environment adaptive dynamic window method is defined as:

[0088] G(v, ω) = a · h(v, ω) + β · c(v, ω) + γ · v(v, ω) + δ · d(v, ω) + λ · e(v, ω);

[0089] where v: linear velocity command of the UAV, controls the speed of the UAV forward or backward;

[0090] ω: angular velocity command of the UAV, controls the speed and direction of the UAV turning;

[0091] h(v, ω): heading component, represents the angle difference between the forward direction of the UAV and the target, measures the degree of the UAV towards the target;

[0092] c(v, ω): clearance component, represents the distance from the nearest obstacle to the current considered curvature of the UAV, measures the safety of the motion;

[0093] v(v, ω): velocity component, represents the translation speed of the UAV, measures the efficiency of the motion;

[0094] d(v, ω): curvature distance component, represents the nearest distance from the target point to the current considered curvature, measures the relevance of the path to the target;

[0095] e(v, ω): dynamic environment adaptability function component, represents the adaptability of the current speed command (v, ω) to the predicted dynamic changes of the environment, measures whether the UAV can effectively respond to the expected motion of dynamic obstacles in the environment when executing the current speed command;

[0096] a, β, γ, δ, λ: weights corresponding to each component;

[0097] Curvature distance component d(v, ω), represents the nearest distance from the target point to the current considered curvature, measures the relevance of the path to the target, the expression is as follows:

[0098]

[0099] β: proximity gain coefficient (0~1), increase the reward when the target is very close to the curvature;

[0100] λ: proximity sensitivity, controls the steepness of the proximity reward;

[0101] δ: direction penalty factor, considers the deviation of the curvature line direction and the target direction;

[0102] θ curv : tangent direction of the curvature line;

[0103] θ goal : direction of the target point relative to the UAV;

[0104] robot_radius: radius of the drone, used to normalize distance values;

[0105] distance_to_goal: shortest distance between the motion curvature produced by the current velocity combination (v, w) and the goal point, is the truncation parameter;

[0106] max_dist: maximum distance threshold, used to normalize and limit the distance influence range.

[0107] In some embodiments, the steps of implementing the AC-time-varying environment adaptive dynamic window method include the following steps:

[0108] Obtain historical observation data and preprocess, convert historical observations to the current drone coordinate system, align historical observations at different times to the same reference coordinate system of the current drone, and define the coordinate system conversion formula as follows:

[0109]

[0110] Where t: current time;

[0111] τ: general time offset parameter, including k, l or g, representing the time difference;

[0112] The converted distance measurement value represents the distance measurement of the historical observation at t-τ time in the coordinate system at the current time t;

[0113] At t-τ time, the distance measurement value with the t-τ time drone coordinate system as the reference;

[0114] The coordinate transformation matrix from t-τ time to t time represents the relative motion speed;

[0115] Θ t : set of state estimates of dynamic objects detected in the environment;

[0116] η: mixing coefficient, dynamically adjusted with time and observation quality Control the weight of the two conversion methods;

[0117] The time derivative of the coordinate transformation represents the relative motion speed.

[0118] ψ(): evaluation function, used to calculate the mixing coefficient η;

[0119] σ(): Swish activation function;

[0120] cart(): function to convert distance scan data in polar form to Cartesian coordinate points;

[0121] F() : forward projection function based on velocity prediction, considering trajectories of moving obstacle candidates including other aircrafts;

[0122] pol() : function to convert points back to polar form from distance scan form;

[0123] Define the neural network structure in the AC-time-varying environment adaptive dynamic window method:

[0124] 1D four-channel input;

[0125] The feature extraction layer processes spatially related scan data;

[0126] The convolutional layer uses the Swish activation function;

[0127] Feature fusion flattens the convolutional output into a vector, connecting target distance, angle, and current speed;

[0128] In the Actor-Critic architecture, the Actor and Critic networks share the same basic structure, except for the last layer;

[0129] Define the training and output decision in the AC-time-varying environment adaptive dynamic window method:

[0130] Three-stage training of different difficulty levels;

[0131] The Actor network inputs system state information, including historical observations, target positions, and current speeds, and outputs actions, including five TEADWA weight values: heading component weight α, gap component weight β, speed component weight γ, curvature distance component weight δ, and dynamic environment adaptability function component weight λ.

[0132] The Critic network outputs state value estimation V(s), which guides the improvement of the Actor through the advantage function.

[0133] To achieve the above purpose, another aspect of the embodiment of the present application proposes a UAV patrol path decision device in a complex environment, the device comprises:

[0134] A graph construction unit for constructing an undirected topological graph with each to-be-patrolled position as a node;

[0135] A steady-state distribution generation unit for generating a steady-state distribution of a UAV patrolling each node according to the topological constraints and node importance of the undirected topological graph;

[0136] a transition matrix generating unit configured to generate a plurality of candidate state transition matrices with the steady-state distribution, wherein each of the candidate state transition matrices comprises transition probabilities of the UAV between each of the nodes;

[0137] an environment assessment unit configured to perform dynamic environment assessment on obstacles of a path corresponding to each of the candidate state transition matrices, so as to determine a selectable speed and direction of the UAV on the selected path;

[0138] a command generating unit configured to dynamically adjust weights of the decision network according to the dynamic environment assessment result, and further generate an execution motion command by using the decision network with the adjusted weights;

[0139] a movement control unit configured to control movement of the UAV between each of the to-be-patrolled positions according to the execution motion command.

[0140] To achieve the above object, another aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0141] To achieve the above object, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0142] The embodiments of the present application at least have the following beneficial effects:

[0143] The present application constructs an undirected topological graph with each to-be-patrolled position as a node; generates a steady-state distribution of the UAV patrolling each of the nodes according to a topological constraint and node importance of the undirected topological graph; performs dynamic environment assessment on obstacles of a path corresponding to each of the candidate state transition matrices, which comprehensively evaluates safety and effectiveness of the UAV speed command in the dynamic environment by coupling autocorrelation score, time proximity score and bias score functions, and quantifies environmental adaptability by weighted sum of risk of each obstacle, so as to determine a selectable speed and direction of the UAV on the selected path; dynamically adjusts weights of the decision network according to the dynamic environment assessment result, and further generates an execution motion command by using the decision network with the adjusted weights; controls movement of the UAV between each of the to-be-patrolled positions according to the motion command, so that the UAV system can dynamically adjust behavior of the time-varying environment adaptive dynamic window approach (TEADWA) according to the environmental state, and obtain adaptability and safety guarantee in the time-varying partially observable environment, especially when the system faces a bird, other UAV / aircraft or unpredictable moving obstacle, to make a safer and more efficient decision. BRIEF DESCRIPTION OF DRAWINGS

[0144] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only represent some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

[0145] Figure 1 A flowchart of a UAV patrol path decision method in a complex environment provided by an embodiment of the present application;

[0146] Figure 2 A Markov chain topology example diagram provided by an embodiment of the present application;

[0147] Figure 3 A MEDRO algorithm flowchart provided by an embodiment of the present application;

[0148] Figure 4 A structure diagram of a UAV patrol path decision device in a complex environment provided by an embodiment of the present application;

[0149] Figure 5 A hardware structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0150] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, but is only an example of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0151] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0152] As used herein, the terms "at least one", "multiple", "each", "any", and the like, include one, two, or more, multiple includes two or more, each refers to each of the corresponding plurality, and any refers to any of the plurality.

[0153] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification herein is for describing the embodiments of the application only and is not intended to be limiting of the application.

[0154] Before the embodiments of the application are described in detail, some related technologies involved in the embodiments of the application are first described as follows:

[0155] DWA: Dynamic Window Approach, a dynamic window method.

[0156] Actor-Critic: a reinforcement learning method, actor-critic.

[0157] DRAMSM: Diverse Random Adaptive Markov Stationary Matrix, a diverse random adaptive Markov stationary matrix.

[0158] DEAF: Dynamic Environment Adaptability Function, a dynamic environment adaptability function.

[0159] TEADWA: Time-varying Environment Adaptive Dynamic Window Approach, a time-varying environment adaptive dynamic window method.

[0160] MoEMamba: Mixture of Experts+Mamba, a network (Mixture of Experts+Mamba) defined by the inventors of the application, used to replace the recurrent neural network in Actor-Critic.

[0161] AC-TEADWA: Actor-Critic Time-varying Environment Adaptive Dynamic Window Approach, AC-time-varying environment adaptive dynamic window method.

[0162] Reference Figure 1The embodiment of the application provides a method for making a patrol path decision of a UAV in a complex environment, which can include but is not limited to S100 to S130, and specifically as follows:

[0163] S100: constructing a non-directional topological graph by taking each to-be-patrolled position as a node;

[0164] S110: generating a steady-state distribution of a UAV patrolling each node according to a topological constraint of the non-directional topological graph and a node importance;

[0165] S120: generating a plurality of candidate state transition matrices with the steady-state distribution; wherein each candidate state transition matrix includes a transition probability of the UAV between each node;

[0166] S130: performing dynamic environment evaluation on an obstacle of a path corresponding to each candidate state transition matrix, so as to determine a selectable speed and direction of the UAV on the selected path;

[0167] S140: dynamically adjusting a weight of a decision network according to a dynamic environment evaluation result, and then generating an execution motion command by using the decision network with the adjusted weight;

[0168] S150: controlling the UAV to move between each to-be-patrolled position according to the execution motion command.

[0169] Optionally, the generating of the steady-state distribution of the UAV patrolling each node according to the topological constraint of the non-directional topological graph includes the following steps:

[0170] The steady-state distribution is generated according to the topological constraint of the non-directional topological graph to improve a probability of the UAV transferring to a target node; wherein, therefore, a high-value target node and a node with a short attack time need to obtain more patrol resources, and the probability of the patrol UAV accessing these nodes is higher;

[0171] An expression of the steady-state distribution is as follows:

[0172]

[0173] Wherein, v i ∈V, v i represents an i-th node, V represents a set of each node, w i represents a value of a target node v i , a i represents an attack time, μ i represents an importance weight parameter of a node v i , μ j represents an importance weight parameter of a node v j , and n is a total number of nodes. representative target node v i a steady state distribution, the target steady state distribution representing a probability distribution of the patrollers at each target node after the Markov chain runs for a long time.

[0174] Optionally, the generating a plurality of candidate state transition matrices with the steady state distribution comprises the following steps:

[0175] According to the multi-element random adaptive Markov steady state matrix algorithm, a plurality of candidate state transition matrices with the steady state distribution and the target network topology are generated, comprising the following steps:

[0176] First stage: hierarchical random matrix generation;

[0177] A first stage random matrix satisfying the topological constraint of the undirected topology graph is generated, wherein the self-loop probability p ii is allocated using four distribution intervals ij ;

[0178] Second stage: relative error iterative adjustment algorithm;

[0179] The first stage random matrix is adjusted according to the error between the probability distribution of the first stage random matrix and the steady state distribution, and specifically, a second stage random matrix is obtained through relative error calculation, error proportion adjustment, row normalization and random periodic disturbance;

[0180] Third stage: similarity value elimination algorithm;

[0181] The second stage random matrix in which the similarity value is greater than a preset similarity threshold value is eliminated, and specifically, a third stage random matrix is obtained by ensuring that there are no too close values in the matrix through similarity value detection, random adjustment and row normalization, enhancing randomness and creating a more natural random distribution;

[0182] Fourth stage: matrix diversity guarantee algorithm;

[0183] The third stage random matrix is adjusted according to the average absolute error between the probability distributions of the third stage plurality of random matrices, and specifically, a fourth stage random matrix is obtained as the candidate state transition matrix by ensuring that the plurality of Markov state transition matrices P k generated have sufficient diversity, rather than being variants of similar matrices.

[0184] Optionally, the dynamic environment assessment of the obstacles corresponding to the path of each candidate state transition matrix comprises the following steps:

[0185] The dynamic environmental adaptability function is used to dynamically evaluate the obstacles on the path corresponding to each candidate state transition matrix, and the risk weighted sum of each obstacle is quantified to quantify the environmental adaptability.

[0186] The expression of the dynamic environmental adaptability function (DEAF) is:

[0187]

[0188] Wherein, e(v, w): dynamic environmental adaptability function, representing the environmental adaptability evaluation value of the unmanned aerial vehicle under the given speed instruction;

[0189] v: linear speed command of the unmanned aerial vehicle, controlling the speed of the unmanned aerial vehicle forward or backward;

[0190] w: angular speed command of the unmanned aerial vehicle, controlling the speed and direction of the unmanned aerial vehicle turning;

[0191] The current system state estimated using weighted observation data contains an environmental representation of obstacle position and speed, wherein z i is the observation data of the unmanned aerial vehicle at the i-th time, and the observation weight is calculated by the following expression:

[0192]

[0193] The weight of the observation data z i quantifies the importance of the set observation in state estimation;

[0194] w a , w r , w d are memory weight coefficients for autocorrelation, time proximity and deviation, starting from initial empirical values, setting according to environmental properties, and updating memory weights regularly using online adaptive method;

[0195] A s (z i , Θ): autocorrelation score of observation, reflecting the time correlation of environmental dynamics;

[0196] Θ: parameter set for controlling autocorrelation score calculation;

[0197] R s (z i ): time proximity score of observation, emphasizing the importance of recent data;

[0198] D s (z i , ψ): deviation score of observation, capturing abnormal changes in the environment;

[0199] Ψ: bias sensitivity parameter, controls the importance of bias to the algorithm;

[0200] M: the number of dynamic obstacles currently perceived, including other UAVs;

[0201] The probability of obstacle i existing calculated based on state estimation Ω, the value range is [0, 1], the larger the value, the higher the collision risk, which measures the probability estimation of obstacle i intersecting with the UAV path when the UAV executes the speed command (ν, ω);

[0202] The relative speed vector of the UAV and obstacle i calculated based on the UAV speed command (ν, ω) and the speed information of the obstacle in the state estimation Ω, which measures the speed and direction of the relative motion of the UAV and the obstacle;

[0203] d i (ν, ω, Ω): the predicted minimum distance between the UAV and obstacle i when the UAV adopts the speed (ν, ω), the smaller the value, the higher the potential collision risk;

[0204] K: normalization constant, ensures the value range of the score item is appropriate, determined based on environmental characteristics and UAV dynamics parameters experiments;

[0205] ε: a positive number, a safety parameter for preventing division by zero.

[0206] Optionally, the step of dynamically adjusting the weight of the decision network according to the dynamic environment evaluation result comprises the following steps:

[0207] The step of dynamically adjusting the weight of the decision network according to the path corresponding to the target state transition matrix system state comprises the following steps:

[0208] Designing the system state, output action and reward function required by the adjustment decision network Actor-Critic;

[0209] Adding the dynamic environment adaptability function to the traditional DWA to obtain the time-varying environment adaptive dynamic window method;

[0210] Combining the decision network Actor-Critic and the time-varying environment adaptive dynamic window method to obtain the AC-time-varying environment adaptive dynamic window method, using the state value estimation V(s) output by the Critic network in the decision network to adjust the weight of the Actor network in the decision network through the advantage function, that is, the weight parameter in the time-varying environment adaptive dynamic window method;

[0211] The Actor-Critic method is replaced by a MoEMamba network in the traditional Actor-Critic method, which is a network that combines and mixes expert networks and Mamba networks, and alternately uses the two networks, one layer of mixed expert networks, one layer of Mamba networks, and so on, a total of k layers;

[0212] System state s t All information contained in the state of the UAV at time t is described, which is the basis of the AC-TEADWA method, and provides complete information required for the neural network to predict AC-TEADWA weights, and the expression of the system state is:

[0213]

[0214] Wherein, s t : The system state at time t contains complete state information of the UAV at the tth moment;

[0215] The distance measurement value of the UAV to the obstacle at the current time t in the current UAV coordinate system;

[0216] The distance measurement value before k control periods represented in the current UAV coordinate system;

[0217] The distance measurement value before l control periods represented in the current UAV coordinate system;

[0218] The distance measurement value before g control periods represented in the current UAV coordinate system;

[0219] d t : The distance from the current position of the UAV to the target;

[0220] φ t : The angle from the current position of the UAV to the target;

[0221] ν t : The current translation speed of the UAV;

[0222] ω t : The current rotation speed of the UAV;

[0223] The change of the obstacle distance between consecutive time steps, capturing dynamic information;

[0224] The time derivative of the distance from the UAV to the target, indicating whether it is approaching the target;

[0225] UAV current acceleration state vector is the linear acceleration of the UAV at the current time t, is the angular acceleration of the UAV at the current time t;

[0226] ξ t : environmental complexity index, calculated by the entropy value of obstacle distribution;

[0227] Specific form of distance measurement value:

[0228] τ represents the past time step, k, l, g in the formula;

[0229] The superscript t-τ represents the measurement time;

[0230] The subscript t represents the coordinate system of the current UAV position;

[0231] N represents the number of measurement points of the laser radar, which is the distance between the UAV position and the nearest obstacle in the set direction;

[0232] The system state is input into the decision network to obtain the action output by the decision network, and the action is expressed as:

[0233] a t =(α t ,β t ,γ t ,δ t ,λ t );

[0234] Wherein, α t : heading term weight;

[0235] β t : gap term weight;

[0236] γ t : speed term weight;

[0237] δ t : curvature distance component, additional weight of the new extended cost function;

[0238] λ t : dynamic environment adaptability function component, additional weight of the new extended cost function;

[0239] The UAV receives the observation or state of the environment and outputs an action a, which is to set the cost function weight of the next control period. These weights will be used to generate the motion command of the UAV;

[0240] Design of reward function:

[0241] ​The UAV successfully reaches the target position, and a larger positive reward R1 is given; the UAV stays at the same position for multiple control periods, similar to the UAV being stuck, and a negative reward R2 is given; the UAV collides with obstacles / or other UAVs / birds during movement, and a larger penalty, negative R3, is given; the UAV is collided by moving obstacles or other UAVs when stationary, and a negative reward R4 is given; the design of such a reward function guides the reinforcement learning algorithm to learn a safe and effective navigation strategy.

[0242] Optionally, the step of implementing the time-varying environment adaptive dynamic window method comprises the following steps:

[0243] The dynamic environment adaptability function is added to the traditional DWA basis to obtain the time-varying environment adaptive dynamic window method;

[0244] The objective function of the time-varying environment adaptive dynamic window method is defined as:

[0245] G(ν,ω)=α·h(ν,ω)+β·c(ν,ω)+γ·v(ν,ω)+δ·d(ν,ω)+λ·e(ν,ω);

[0246] h(ν,ω): heading component, representing the angle difference between the UAV's forward direction and the target, measuring the degree of the UAV towards the target;

[0247] c(ν,ω): gap component, representing the distance from the UAV to the nearest obstacle at the current considered curvature, measuring the safety of the motion;

[0248] v(ν,ω): velocity component, representing the translational speed of the UAV, measuring the efficiency of the motion;

[0249] d(ν,ω): curvature distance component, representing the nearest distance from the target point to the current considered curvature, measuring the relevance of the path to the target;

[0250] e(ν,ω): dynamic environment adaptability function component, representing the adaptability of the current speed command (ν,ω) to the predicted environmental dynamic changes, measuring whether the UAV can effectively respond to the expected motion of the dynamic obstacles in the environment when executing the current speed command;

[0251] α, β, γ, δ, λ: weights corresponding to each component;

[0252] The curvature distance component d(ν,ω) represents the nearest distance from the target point to the current considered curvature, measures the relevance of the path to the target, and the expression is as follows:

[0253]

[0254] Alpha: Distance decay factor (≥ 1), controls the degree of nonlinearity of distance impact;

[0255] Beta: Proximity gain factor (0~1), increases the reward when the target is very close to the curvature;

[0256] Lambda: Proximity sensitivity, controls the steepness of the proximity reward;

[0257] Delta: Directional penalty factor, considers the deviation of the curvature line direction from the target direction;

[0258] Theta curv : Tangent direction of the curvature line;

[0259] Theta goal : Direction of the target point relative to the UAV;

[0260] Robot_radius: Radius of the UAV, used to normalize the distance value;

[0261] Distance_to_goal: The shortest distance between the motion curvature generated by the current speed combination (v, w) and the target point, which is the truncation parameter;

[0262] Max_dist: Maximum distance threshold, used for normalization and limiting the distance impact range.

[0263] Optionally, the steps of implementing the AC-time-varying environment adaptive dynamic window method include the following steps:

[0264] Obtain historical observation data and preprocess, convert historical observations to the current UAV coordinate system, align historical observations at different times to the same reference coordinate system of the current UAV, and define the coordinate system conversion formula as follows:

[0265]

[0266] Where, t: current time;

[0267] Tau: General time offset parameter, including k, l or g, representing the time difference;

[0268] The converted distance measurement value represents the distance measurement of the historical observation at t-τ time in the coordinate system at the current time t;

[0269] At t-τ time, the distance measurement value with t-τ time UAV coordinate system as reference;

[0270] Coordinate transformation matrix from t-τ time to t time, representing the relative motion speed;

[0271] Thetat : Set of state estimates of dynamic objects detected in the environment

[0272] η: Mixing coefficient, dynamically adjusted with time and observation quality Control the weight of the two conversion methods

[0273] Time derivative of coordinate transformation, representing relative motion speed.

[0274] ψ(): Evaluation function for calculating mixing coefficient η

[0275] σ(): Swish activation function

[0276] cart(): Function to convert distance scan data in polar form to Cartesian coordinate points

[0277] F(): Forward projection function based on velocity prediction, considering trajectory of moving obstacle candidates including other aircraft

[0278] pol(): Function to convert points back to distance scan form in polar form

[0279] Define the neural network structure in the AC-time-varying environment adaptive dynamic window method:

[0280] 1D four-channel input

[0281] Feature extraction layer processes spatially related scan data

[0282] Convolutional layer uses Swish activation function

[0283] Feature fusion flattens convolutional output into a vector, connecting target distance, angle, and current speed

[0284] In the Actor-Critic architecture, Actor and Critic networks share the same basic structure, only the last layer is different

[0285] Define the training and output decision in the AC-time-varying environment adaptive dynamic window method

[0286] Three-stage training of different difficulty levels

[0287] Actor network input system state information, including historical observations, target position, and current speed, output action, including five TEADWA weight values: heading component weight α, gap component weight β, speed component weight γ, curvature distance component weight δ, and dynamic environment adaptability function component weight λ

[0288] The critic network outputs a state value estimate V(s) to guide the improvement of the actor through a advantage function.

[0289] Next, the scheme of the embodiment of the present application will be described in detail with reference to specific application examples.

[0290] Specifically, the embodiment includes the following technical solutions:

[0291] The embodiment proposes a novel unmanned aerial vehicle intelligent patrol path decision system and method (DRAMSM-AC-TEADWA): two algorithms are connected through a system environment module, and the system environment includes information such as network topology, steady-state conditions, and obstacle constraints.

[0292] First, a diverse random adaptive Markov stationary matrix (DRAMSM) algorithm is developed, which is a four-stage combinatorial optimization algorithm. On the basis of meeting the target network topology structure, the algorithm generates a plurality of Markov state transition matrices (path selection) through optimization processes such as layer random matrix generation, relative error iterative adjustment, similarity value elimination, and matrix diversity guarantee. The generated Markov state transition matrices not only have sufficient diversity but also meet the special requirements of the target steady-state distribution.

[0293] Second, a dynamic environment adaptability function (DEAF) is developed to handle dynamic obstacles in complex environments. The function evaluates the safety and effectiveness of the unmanned aerial vehicle speed command in the dynamic environment by coupling autocorrelation scores, time proximity scores, and bias score functions. The environment adaptability is quantified by weighting and summing the risk of each obstacle.

[0294] Finally, the AC-TEADWA (Actor-Critic Time-varying Environment Adaptive Dynamic Window Approach) algorithm was developed. By incorporating the Dynamic Environment Adaptive Function (DEAF) into the traditional Dynamic Window Approach (DWA), the TEADWA algorithm was formed, enabling the system to more accurately estimate the current environmental state. The weights of the objective function were adjusted as the action space for reinforcement learning. The Actor-Critic (MoEMamba network) algorithm was used to adjust the TEADWA parameters, resulting in the final AC-TEADWA algorithm. This allows the UAV system to dynamically adjust the behavior of TEADWA according to the environmental state, achieving both adaptability and safety in time-varying partially observable environments. In particular, when the system faces birds, other UAVs / aircraft, or unpredictable moving obstacles, it makes safer and more efficient decisions.

[0295] More specifically, the specific implementation of this embodiment will be described below.

[0296] 1. System modeling.

[0297] The physical environment is discretized based on the actual patrol situation of the drone and modeled using an undirected topological graph, G = {V, E}, where V = {v1, v2, ..., v n} represents the set of vertices (drone patrol locations), and each node v in the graph... i Represents locations in the real-world environment that require drone patrol and surveillance, E = {(i,j)|i,j∈V,p} ij The set of edges of the table p ≥ 0, i.e., the set of paths between two patrol locations. ij >0 represents a node v i Transfer to node v j The probability that the drone actually flies between the two patrol locations satisfies the following conditions. p ij With p ji They are not necessarily equal. ij Representative node v i and v j Connected to each other, Representative node v i and v j Path length between, w i ∈R + Representative node v i The value of a i ∈R +The attacker (the attacking drone) attacked node v i The time required. Figure 2 Let P represent a Markov chain with five nodes and its corresponding transition matrix P, where P is irreducible and aperiodic. An example of a state transition matrix is ​​shown below:

[0298]

[0299] The transition probabilities between nodes are encoded in the transition matrix P. Each row of matrix P represents a probability distribution (i.e., all values ​​in each row add up to 1). The patrol drone can start from any initial vertex (initial patrol position) and then sample the probability distribution in the row corresponding to its current vertex to determine which area should be patrolled / inspected next.

[0300] 2. Multivariate stochastic adaptive Markov steady-state matrix (DRAMSM) algorithm.

[0301] The Diverse Random Adaptive Markov Stationary Matrix (DRAMSM) algorithm is a four-stage combinatorial algorithm that efficiently solves the problem of generating stochastic Markov state transition matrices with specific constraints by generating steady-state distributions that meet specific requirements, based on the target network topology.

[0302] 2.1 Generate the optimal steady-state distribution.

[0303] like Figure 2 The Markov chain shown has different nodes v i Representing different locations requiring drone patrols, different nodes (patrol locations) have different values, and attackers need different amounts of time to attack different nodes. Therefore, high-value target nodes and nodes with short attack times require more patrol resources, and patrol drones are more likely to visit these nodes. π=[π1,π2,…,π n ] is a general steady-state distribution, representing the probability distribution of the patroller at each target point after the Markov chain has been running for a long time. Therefore, it is necessary to design a steady-state distribution with specific optimization properties. Formula (1) shows how to calculate the steady-state distribution considering the target value and attack time.

[0304]

[0305] in, v i ∈V, v i Let V represent the i-th node, and let w represent the set of all the nodes. i Represents the target node v ithe value of a i representing the attack time, μ i representing the node v i the importance weight parameter of v j representing the node v j the importance weight parameter of v, n is the total number of nodes, representing the target node v i the steady-state distribution of the target node v, the target steady-state distribution represents the probability distribution of the patroller at each target node after the Markov chain runs for a long time.

[0306] 2.2, DRAMSM algorithm generates transition matrix.

[0307] In order to increase the prediction difficulty of the patrol unmanned aerial vehicle, a plurality of transition matrices P with the same steady-state distribution π * (formula 1) but different transition characteristics are generated by using a multi-element random adaptive Markov steady-state matrix (DRAMSM) algorithm, so as to realize a random path selection strategy of space-time decoupling, as shown in Figure 3 Specifically, the multi-element random adaptive Markov steady-state matrix (DRAMSM) algorithm has four stages respectively responsible for hierarchical random matrix generation, relative error iterative adjustment, multi-similarity value elimination and matrix diversity guarantee, and the matrix is continuously improved by iteration to meet the constraints of network topology, line sum and steady-state conditions.

[0308] First stage: hierarchical random matrix generation.

[0309] 1. Self-loop probability generation:

[0310] For each node i, the self-loop probability p ij is assigned according to the self-loop type using four distribution intervals.

[0311] 2. Non-self-loop transition probability generation:

[0312] Determine the connectable node set C i of each node i (based on the adjacency matrix), calculate the remaining probability, generate a weight vector w using three distributions, add random disturbance, normalize the weight sum and assign the transition probability.

[0313] This stage constructs an initial random matrix Q, which satisfies the network topology constraint as shown in Figure 2 , but usually does not satisfy the predefined steady-state distribution π *

[0314] Second stage: relative error iterative adjustment algorithm.

[0315] 1. Steady-state distribution verification:

[0316] Calculate the steady-state effect of the current matrix P and calculate the L1 error.

[0317] 2. Relative error calculation:

[0318] For each state j, calculate the relative error, positive value means increase the transition to this state, negative value means decrease the transition to this state.

[0319] 3. Error proportion adjustment:

[0320] Set a decay learning rate, calculate the adjustment factor, apply multiplication adjustment to each valid transition probability to ensure the minimum probability threshold.

[0321] 4. Row normalization:

[0322] Calculate the sum of each row and normalize to ensure the Markov property.

[0323] 5. Periodic random disturbance:

[0324] Add a random disturbance of ±5% every 10 iterations.

[0325] This stage adjusts the initial matrix Q through iterations to make its steady-state distribution close to the target distribution π * , the calculation process is similar to the gradient descent method with momentum and decay learning rate, but the object of operation is the transition matrix instead of the parameter vector in the standard optimization problem.

[0326] Third stage: Similar value elimination algorithm.

[0327] 1. Similar value detection:

[0328] Sort the valid elements in each row i (i.e. ij >0 and topologically allowed), check the difference between adjacent sorted value pairs, if the difference is less than the threshold, it is determined as a similar value.

[0329] 2. Random adjustment:

[0330] Randomly select one value in the similar pair to generate an adjustment amount, randomly decide to increase or decrease the adjustment amount.

[0331] 3. Row normalization:

[0332] Renormalize the row after processing, keep the probability sum to 1.

[0333] This stage ensures that there are no too close values in the matrix, enhances randomness, and creates a more natural random distribution.

[0334] Fourth stage: Matrix diversity guarantee algorithm.

[0335] 1. Matrix difference calculation:

[0336] Calculate the difference between any two matrices P a and Pb the average absolute error between them.

[0337] 2. Diversity check:

[0338] If the average absolute error is less than the error diversity threshold, the two matrices are considered too similar and need to be adjusted.

[0339] 3. Large random perturbation:

[0340] Randomly select one of the matrices and apply a large random perturbation of ±10%, then normalize.

[0341] 4. Re-adjust the steady-state distribution:

[0342] Re-apply the second-stage iterative adjustment algorithm to the perturbed matrix to ensure that the adjusted matrix still satisfies the steady-state distribution * requirements.

[0343] This stage ensures that the generated multiple Markov state transition matrices P k have sufficient diversity and are not variants of similar matrices.

[0344] 3. Dynamic environment adaptability function.

[0345] The dynamic environment adaptability function (DEAF) comprehensively evaluates the safety and effectiveness of the UAV speed command in the dynamic environment, and quantifies the environment adaptability by weighted sum of the risk of each obstacle. The function value close to 1 indicates that the speed command is safe and adaptable to environmental changes; close to 0 indicates that it may collide with dynamic obstacles and is not adaptable to the current environmental state, as shown in equation (2).

[0346]

[0347] where e(v, ω): dynamic environment adaptability function, represents the environment adaptability evaluation value of the UAV under the given speed command;

[0348] v: linear speed command of the UAV, controls the speed size of the UAV forward or backward;

[0349] ω: angular speed command of the UAV, controls the speed size and direction of the UAV turning;

[0350] The current system state estimated using weighted observation data, containing the environment representation of obstacle position and speed, where z i is the observation data of the UAV at the i-th time, and the observation weight is calculated by the following expression:

[0351]

[0352] observed data z i weights, quantifying the importance of the set observations in state estimation;

[0353] w a , w r , w d are memory weight coefficients for auto-correlation, temporal proximity, and bias, starting from initial empirical values, periodically updated using online adaptive methods according to environmental properties settings;

[0354] A s (z i , Θ): auto-correlation score of observations, reflecting the temporal correlation of environmental dynamics;

[0355] Θ: parameter set controlling auto-correlation score calculation;

[0356] R s (z i ): temporal proximity score of observations, emphasizing the importance of recent data;

[0357] D s (z i , ψ): bias score of observations, capturing abnormal changes in the environment;

[0358] ψ: bias sensitivity parameter, controlling the algorithm's emphasis on bias;

[0359] M: current perceived number of dynamic obstacles, including other UAVs;

[0360] Obstacle i existence probability calculated based on state estimation Ω, with a value range of [0, 1], the larger the value, the higher the collision risk, measuring the probability estimate of obstacle i intersecting with the UAV path when the UAV executes speed command (v, ω);

[0361] Relative speed vector between UAV and obstacle i calculated based on UAV speed command (v, ω) and obstacle speed information in state estimation Ω, measuring the speed and direction of relative motion between UAV and obstacle;

[0362] d i (v, ω, Ω): predicted minimum distance between UAV and obstacle i when adopting speed (v, ω), the smaller the value, the higher the potential collision risk;

[0363] K: normalization constant, ensuring the value range of score items is appropriate, determined based on environmental characteristics and UAV dynamics parameters experiments;

[0364] ε: a positive number to prevent division by zero, taking values in the interval [10 -6 -3

[0365] 3.1, Autocorrelation score.

[0366] The autocorrelation function measures the degree of similarity between time points in a time series, with high autocorrelation values indicating that the environment changes have a certain regularity or periodicity, making past observations useful for predicting future states. In a dynamic moving obstacle environment, capturing the temporal correlation patterns of the environment dynamics, the system can identify which historical observation patterns are similar to the current situation, thus prioritizing those observations that are more likely to provide useful information, enabling the system to predict the future position of the obstacles, as shown in equation (4).

[0367]

[0368] where A s (z i , Θ): the integrated autocorrelation score of the observation z i at time t i , controlled by the parameter set Θ;

[0369] z i : the observation at the jth time point in the time series, representing the observation of the obstacle position by the UAV's lidar ranging, position estimation, or sensors;

[0370] z j : the observation at the jth time point in the time series;

[0371] the observation at the j+|t-t i |th time point in the time series;

[0372] the mean of all observations, representing the baseline or reference value of the observations, reflecting the average level of the environment state;

[0373] t: the current time;

[0374] t i : the observation time point i;

[0375] |t-t i | : the time difference between the current time t and the observation time ti, as the lag quantity of autocorrelation;

[0376] ω j : the correlation weight at time point j, emphasizing the contribution of observations at certain time points to the correlation calculation;

[0377] ​​ and The weight coefficient of observations at different time points can be used to handle missing data or uncertain observations;

[0378] λ: time decay parameter, j0 is the reference time point, so that the farther away from the reference point, the smaller the contribution of the pair correlation;

[0379] γ: time lag penalty coefficient, so that the greater the time difference, the faster the score decay;

[0380] δ: periodicity impact factor, reflecting the periodic changes in the environment (such as tides, day-night cycle);

[0381] T cycle : the period length of environmental changes;

[0382] Parameter set controlling the calculation of autocorrelation score;

[0383] n: observation sequence length, representing the size of the historical data window considered.

[0384] The system maintains an observation sequence {z1,z2,…,z n}, the current time is t, and the historical observation z i occurs at time t i . First, calculate the mean of the observation sequence z, then for each historical observation z i , calculate the time difference (t-t i ) from the observation to the current time, use the time difference as the lag parameter, evaluate the autocorrelation A s (z i ,Θ) of the entire observation sequence at this lag value.

[0385] 3.2, Time proximity score.

[0386] The time proximity score gives higher weight to more recent observations, reflecting the temporal locality of environmental state in a moving obstacle environment, and recent observations are usually more representative of the current environmental state than long-term observations, as shown in equation (5).

[0387]

[0388] where R s (z i ): the time proximity score of observation z i , quantifying the proximity of observation time to the current time.

[0389] t: current time;

[0390] t i : observation time point i;

[0391] η: Normalization constant to ensure scores are within a reasonable range;

[0392] κ: Time decay exponent controlling the rate of decay of timeliness over time;

[0393] ε: A small positive number to prevent division by zero safety parameter, taking values between 10-6 and 10-3;

[0394] τ d : Exponential decay time constant representing the observed "half-life";

[0395] φ: Additional weight coefficient for recent observations;

[0396] τ c : Critical time window within which observations receive additional weight.

[0397] 3.3, Bias score.

[0398] The bias score identifies observations that significantly deviate from normality, aiding in capturing sudden events or critical turning points in a moving obstacle environment, such as an obstacle suddenly changing direction or speed, alerting the system to this potential important state transition, as shown in equation (6).

[0399]

[0400] where D s (z i , ψ): Parameterized bias score for observation z

[0401] z i : Observation value at the jth time point in the time series;

[0402] ψ: Bias sensitivity parameter controlling the algorithm's emphasis on bias;

[0403] p: Bias power, increasing the degree of nonlinearity;

[0404] Standard deviation of observations used to normalize bias;

[0405] ξ: Stability constant to prevent division by zero;

[0406] ν: Outlier gain coefficient;

[0407] 1(·): Indicator function, 1 if condition is met, otherwise 0;

[0408] χ: Outlier threshold coefficient defining when an observation is considered an outlier;

[0409] The average of all observations within the current time window is considered.

[0410] 4. AC- Time-varying Environment Adaptive Dynamic Window Approach (ACTEADWA).

[0411] In order to make the UAV better track the selected / planned path, the algorithm combining deep reinforcement learning (Actor-Critic) and time-varying environment adaptive dynamic window approach (TEADWA) is adopted. The Actor (MoEMamba) network outputs the action (five TEADWA weight values) based on the input environment state (historical observations, target position, current speed), and the Critic (MoEMamba) network outputs the state value estimate V(s). The improvement of the Actor (policy) is guided by the advantage function, so as to realize a more stable training process. The objective function of AC- Time-varying Environment Adaptive Dynamic Window Approach (AC-TEADWA) uses artificial intelligence to adjust the parameters of the TEADWA method, instead of directly generating motion commands, which not only obtains the adaptability of deep learning, but also maintains the safety guarantee of traditional DWA.

[0412] 4.1. Setting of system-related states.

[0413] Design of system state: system state s t All the information contained in the state of the UAV at time t is described, which is the basis of the AC-TEADWA method and provides the complete information required for the neural network to predict the AC-TEADWA weight, as shown in equation (7).

[0414]

[0415] where s t : the system state at time t, which contains the complete state information of the UAV at the t-th time;

[0416] The distance measurement value of the UAV to the obstacle at the current time t in the current UAV coordinate system;

[0417] The distance measurement value before k control periods represented in the current UAV coordinate system;

[0418] distance measurement before l control periods in current UAV coordinate system;

[0419] distance measurement before g control periods in current UAV coordinate system;

[0420] d t : distance from UAV current position to target;

[0421] φ t : angle from UAV current position to target;

[0422] ν t : current translational velocity of UAV;

[0423] ω t : current rotational velocity of UAV;

[0424] obstacle distance change between consecutive time steps, capturing dynamic information;

[0425] time derivative of UAV-target distance, indicating whether approaching target;

[0426] current acceleration state vector of UAV is linear acceleration at current time t of UAV, is angular acceleration at current time t of UAV;

[0427] ξ t : environmental complexity index, calculated by entropy of obstacle distribution;

[0428] specific form of distance measurement:

[0429] τ represents past time steps, k, l, g in the formula;

[0430] superscript t-τ represents measurement time;

[0431] subscript t represents coordinate system of current UAV position;

[0432] N represents number of measurement points of lidar, distance between UAV position and nearest obstacle in set direction.

[0433] Using observations at multiple time points can help predict the motion of dynamic obstacles, including target information helps to conduct target-oriented navigation, and including current motion state helps to generate smooth control commands.

[0434] Action design: The UAV receives the observation (or state) of the environment, outputs an action a, which is to set the cost function weight for the next control period, these weights will be used to generate the motion command of the UAV.

[0435] a t t , β t , γ t , δ t , λ t ).

[0436] α t : heading term weight.

[0437] β t : gap term weight.

[0438] γ t : speed term weight.

[0439] δ t : curvature distance component, additional weight of the new extended cost function.

[0440] λ t : dynamic environment adaptability function component, additional weight of the new extended cost function.

[0441] This design adjusts the weight of ACTEADWA as the action space of reinforcement learning, so that the system can dynamically adjust the behavior of TEADWA according to the state of the environment.

[0442] Reward function design: the UAV successfully reaches the target position, and gives a larger positive reward R1. The UAV stays in the same position for multiple control periods, similar to the UAV being stuck, and gives a negative reward R2. The UAV collides with obstacles / or other UAVs / birds during movement, and gives a larger penalty, negative R3. The UAV is collided by moving obstacles or other UAVs when it is stationary, and gives a negative reward R4. Through the design of this reward function, the reinforcement learning algorithm is guided to learn a safe and effective navigation strategy.

[0443] 4.2, AC-time-varying environment adaptive dynamic window method (AC-TEADWA) algorithm design.

[0444] ​AC-TEADWA makes TEADWADWA more robust in time-varying partially observable environments, especially when the system faces flying birds, other UAVs or unpredictably moving obstacles. This time, correlation and anomaly-based weight calculation enables the system to more accurately estimate the current environmental state, making safer and more efficient decisions. The objective function is shown in equation (8).

[0445] G(v, ω) = a · h(v, ω) + β · c(v, ω) + γ · v(v, ω) + δ · d(v, ω) + λ · e(v, ω) (8)

[0446] h(v, ω): heading component, representing the angle difference between the UAV's heading direction and the target, measuring the extent to which the UAV is heading towards the target.

[0447] c(v, ω): clearance component, representing the distance from the UAV to the nearest obstacle at the current considered curvature, measuring the safety of the motion.

[0448] v(v, ω): velocity component, representing the translational speed of the UAV, measuring the efficiency of the motion.

[0449] d(v, ω): curvature distance component, representing the nearest distance from the target point to the current considered curvature (i.e. the motion trajectory the UAV will form under a specific speed combination), measuring the relevance of the path to the target.

[0450] e(v, ω): dynamic environment adaptability function component, representing the adaptability of the current speed command (v, ω) to the predicted environmental dynamic changes, measuring whether the UAV can effectively respond to the expected motion of dynamic obstacles in the environment when executing the current speed command.

[0451] a, β, γ, δ, λ: weights corresponding to each component.

[0452] Curvature distance component d(v, ω), representing the nearest distance from the target point to the current considered curvature (i.e. the motion trajectory the UAV will form under a specific speed combination), measuring the relevance of the path to the target, as shown in equation (9).

[0453]

[0454] a: distance decay factor (≥ 1), controlling the degree of nonlinearity of distance influence;

[0455] β: proximity gain coefficient (0-1), increasing the reward when the target is very close to the curvature;

[0456] λ: proximity sensitivity, controls how steep the proximity reward is;

[0457] δ: direction penalty factor, considers the deviation of the curvature line direction from the target direction;

[0458] θ curv : tangent direction of the curvature line;

[0459] θ goal : direction of the goal point relative to the UAV;

[0460] robot_radius: radius of the UAV, used to normalize distance values;

[0461] max_dist: maximum distance threshold, used for normalization and limiting the distance impact range.

[0462] where distance_to_goal is the shortest distance between the motion curvature generated by the current speed combination (v, ω) and the goal point, and is the truncation parameter. This formula skillfully converts the physical distance into a score in the [0, 1] interval, and ensures that the closer the curvature to the target, the higher the score, which meets the goal-oriented needs of navigation. It provides a perspective of "avoiding obstacles" to help the UAV make better decisions in situations where the heading component is not favorable, and solves the problem of the UAV oscillating in front of obstacles.

[0463] Each speed combination (v, ω) in the ACTEADWA search space corresponds to a possible curvature, and the cost function evaluates the "goodness" (safety, proximity to the target, etc.) of each curvature. Through reinforcement learning, the corresponding weights in the cost function are optimized to select the optimal curvature (i.e., the optimal speed combination).

[0464] 4.3, AC-TEADWA algorithm implementation.

[0465] The AC-TEADWA algorithm uses artificial intelligence to adjust the parameters of the traditional DWA method. The Actor (MoEMamba) network outputs actions (five TEADWA weight values) based on input state information (historical observations, target location, current speed), and the Critic (MoEMamba) network outputs state value estimates V(s). Through the advantage function, the improvement of the Actor (policy) is guided, thereby realizing a more stable training process.

[0466] 4.3.1, Acquisition and preprocessing of historical observations.

[0467] 4.3.1.1 Multi-time observation data collection.

[0468] Laser scan data of the drone at four time points, data at the current time (t) Data k control periods ago (t-k) Data l control periods ago (t-l) Data g control periods ago (t-g) N distance values are obtained at each time point.

[0469] 4.3.1.2 Coordinate system conversion.

[0470] Converting historical observations to the current drone coordinate system using formula (10) aligns historical observations at different times to the same reference coordinate system of the current drone, making it possible to directly compare historical observations with current observations, which facilitates the neural network to learn the motion pattern of obstacles.

[0471]

[0472] Where t: current time;

[0473] τ: general time offset parameter (k, l or g), representing the time difference;

[0474] The converted distance measurement value represents the distance measurement of the historical observation at time t-τ in the coordinate system at time t;

[0475] The distance measurement value at time t-τ, with the drone coordinate system at time t-τ as the reference;

[0476] The coordinate transformation matrix from time t-τ to time t, representing the relative motion speed;

[0477] Θ t : Set of state estimates of dynamic objects detected in the environment;

[0478] η: Mixing coefficient, dynamically adjusted over time and observation quality Control the weight of the two conversion methods;

[0479] The time derivative of the coordinate transformation, representing the relative motion speed.

[0480] ψ(): Evaluation function used to calculate the mixing coefficient η;

[0481] σ(): Swish activation function;

[0482] cart(): Function to convert distance scan data (in polar form) to Cartesian coordinate points;

[0483] F(): Forward projection function based on velocity prediction, considering the trajectories of moving obstacles (including other aircraft) candidates;

[0484] pol(): Function to convert points back to range scan form (polar form);

[0485] Step 1:

[0486] Input: is the original range measurement at time t - τ.

[0487] Function: Convert range scan data (polar form) to Cartesian coordinate points.

[0488] Output: A series of Cartesian coordinate points.

[0489] Step 2:

[0490] Input: Cartesian coordinate points from the previous step.

[0491] Coordinate transformation matrix from time t - τ to time t.

[0492] Function: Transform points at time t - τ to the UAV coordinate system at time t.

[0493] Output: Points in the current UAV coordinate system (Cartesian form).

[0494] Step 3: pol(...).

[0495] Input: Cartesian coordinate points in the current coordinate system.

[0496] Function: Convert points back to range scan form (polar form).

[0497] Processing: Use binning and interpolation to handle unknown values.

[0498] Output: Range measurements in the current coordinate system (polar form).

[0499] 4.3.1.3 Data processing and standardization.

[0500] Handling "unknown values": Discretize the angular space using binning techniques and estimate missing values using interpolation.

[0501] Range clipping: Limit all range values within dclip to ensure data range consistency.

[0502] 4.3.2, Neural network input design.

[0503] 4.3.2.1 Input structure.

[0504] Form 1D four-channel input:

[0505] First channel: current observation

[0506] Second channel: observation k control periods ago (converted to current coordinate system).

[0507] Third channel: observation l control periods ago (converted to current coordinate system).

[0508] Fourth channel: observation g control periods ago (converted to current coordinate system).

[0509] 4.3.2.2 Additional state information.

[0510] Target-related information: distance d t and angle φ t .

[0511] UAV motion state: current translational velocity v t and rotational velocity ω t .

[0512] 4.3.3, Network architecture design.

[0513] 4.3.3.1 Feature extraction layer.

[0514] Three layers of one-dimensional convolution: process spatially dependent scan data.

[0515] All convolution layers use Swish activation function.

[0516] 4.3.3.2 Feature fusion.

[0517] Flatten the convolution output into a vector, connect target distance, angle, current speed, etc.

[0518] 4.3.3.3 Decision layer.

[0519] m fully connected layers n four linear neurons, output corresponding DWA weights α, β, γ, δ, λ.

[0520] 4.3.4, Implicit prediction step.

[0521] 4.3.4.1 Time structure learning.

[0522] The network does not explicitly output the future position of the obstacle, but learns the time pattern of observations at three time points through the convolution layer.

[0523] Convolutional operations establish connections between different time points at the same angle, enabling the network to detect distance change patterns (i.e. movement).

[0524] 4.3.4.2 Motion pattern recognition.

[0525] Static obstacles have a position that is essentially fixed in the observation of three time points.

[0526] Dynamic obstacles have a position that changes in the observation of different time points.

[0527] The network learns to recognize these change patterns and predict their trends.

[0528] 4.3.4.3 Weight adaptation mechanism.

[0529] When obstacles are detected to be moving, adjust a (heading weight), β (gap weight), adjust γ (speed weight), δ (distance to curvature weight), λ (dynamic environment adaptability function weight) Different motion patterns correspond to different weight combinations.

[0530] 4.3.5, Training and optimization.

[0531] 4.3.5.1 Reinforcement learning framework.

[0532] Using the Actor-Critic (MoEMamba network) method, the MoEMamba network replaces the recurrent neural network in the traditional Actor-Critic method, and MoEMamba is a new network designed in this paper, which combines and mixes the Mixture of Experts (MoE) and Mamba network. Alternately use two networks, one layer of Mixture of Experts network, one layer of Mamba network, and so on, a total of k layers. The critic network is responsible for evaluating the value of the state, and the improvement of the Actor (policy) is guided through the advantage function, so as to realize a more stable training process.

[0533] Objective function of Actor:

[0534]

[0535] Where, s: system state, input the state information of the environment to the Actor network;

[0536] a: control action output by the Actor network;

[0537] w k : current Actor network parameters (parameters of the kth iteration);

[0538] w: Actor network parameter variable to be optimized;

[0539] π w : current updated policy, policy function based on parameter w;

[0540] experience-generating policy;

[0541] advantage function, measures the advantage of taking action a in state s relative to the average level;

[0542] η: clipping constant, limits the range of policy changes, controls the update step size;

[0543] clip(): clipping function, limits the value to the range [1-η, 1+η];

[0544] Actor updates network parameters:

[0545]

[0546] Actor updates parameters w by maximizing the objective function L k is the current Actor parameter, w k+1 is the updated Actor parameter.

[0547] Critic's advantage function:

[0548]

[0549] where s t : system state at time t, input state information of the environment to the Actor network;

[0550] a t : control action output by the Actor network at time t;

[0551] r t : immediate reward value obtained at time t;

[0552] r t+1 : reward value obtained at time t+1;

[0553] r t+n : reward value obtained at time t+n;

[0554] γ: discount factor, used to attenuate the weight of future rewards, value between 0 and 1;

[0555] γ t+n : (t+n)th power of the discount factor, indicating the degree of discounting of the reward at time t+n.

[0556] Critic's value function:

[0557]

[0558] wherein π w : policy function based on parameter w;

[0559] n: time step, representing the upper limit of the number of time steps considered;

[0560] s t : system state at time t, input state information of the environment to the Actor network;

[0561] r0: reward value at the initial time (0th step);

[0562] r1: reward value at the 1st step;

[0563] r n : reward value at the nth step;

[0564] γ: discount factor, used to attenuate the weight of future rewards;

[0565] γ n : nth power of the discount factor, representing the degree of discounting for the nth step reward;

[0566] Value function of state s w under policy π t , representing the expected cumulative reward of performing according to policy π t starting from state s w ;

[0567] In the Actor-Critic (MoEMamba network) architecture, the Actor network outputs actions (five DWA weight values) based on input state information (history observation, target location, current speed), and the Critic network outputs state value estimation V(s). The Actor and Critic networks share the same basic structure, except for the last layer.

[0568] 4.3.5.2 Three-stage training.

[0569] First stage: no moving obstacles, the UAV first learns basic navigation.

[0570] Second stage: obstacles with the same moving speed, the UAV learns to handle simple dynamic environments.

[0571] Third stage: obstacles with different moving speeds, the UAV learns to handle complex dynamic environments.

[0572] In summary, the embodiment includes the following technical solutions:

[0573] 1. A complex environment intelligent patrol path decision system and method for unmanned aerial vehicles is proposed: Diverse Random Adaptive Markov Stationary Matrix-Actor-Critic Time-varying Environment Adaptive Dynamic Window Approach (DRAMSM-AC-TEADWA), which simultaneously considers random path selection, path tracking, and dynamic obstacle avoidance.

[0574] 2. A Diverse Random Adaptive Markov Stationary Matrix (DRAMSM) algorithm is developed, which generates multiple Markov state transition matrices (path selection) with sufficient diversity while satisfying network topology, probability conservation, and target steady-state distribution constraints through layer random matrix generation, relative error iterative adjustment, similarity value elimination, and matrix diversity guarantee optimization processes.

[0575] 3. A Dynamic Environment Adaptability Function (DEAF) is developed to handle dynamic obstacles in complex environments. This function evaluates the safety and effectiveness of the UAV's speed command in dynamic environments by coupling autocorrelation scores, time proximity scores, and bias score functions. The environmental adaptability is quantified by weighting and summing the risk of each obstacle.

[0576] 4. An Actor-Critic Time-varying Environment Adaptive Dynamic Window Approach (AC-TEADWA) algorithm is developed by incorporating the Dynamic Environment Adaptability Function (DEAF) into the traditional Dynamic Window Approach (DWA) to form the Time-varying Environment Adaptive Dynamic Window Approach (TEADWA). This allows the system to more accurately estimate the current environmental state and adjust the weight of its objective function as the action space for reinforcement learning. The Actor-Critic (MoEMamba network) algorithm is used to adjust the TEADWA parameters, resulting in the final AC-TEADWA algorithm. This enables the UAV system to dynamically adjust the TEADWA behavior based on the environmental state, ensuring adaptability and safety in time-varying partially observable environments. In particular, when the system encounters birds, other UAVs / aircraft, or unpredictable moving obstacles, it makes safer and more efficient decisions.

[0577] The benefits of this embodiment include:

[0578] The embodiment integrates an intelligent unmanned aerial vehicle patrol decision system and method. First, a diverse random adaptive Markov stationary matrix (DRAMSM) algorithm is developed, which is particularly suitable for optimizing a random patrol strategy for a time-varying Markov decision process and generating a highly randomized unmanned aerial vehicle patrol path that is more difficult to predict than a path generated by an existing algorithm.

[0579] Second, a dynamic environment adaptability function (DEAF) is added to the traditional DWA, which comprehensively evaluates the safety and effectiveness of the unmanned aerial vehicle speed command in the dynamic environment by coupling the autocorrelation score, time proximity score and bias score function, and quantifies the environment adaptability by weighted sum of the risk of each obstacle.

[0580] Finally, an AC-time-varying environment adaptive dynamic window method (AC-TEADWA) is proposed by combining an Actor-Critic (MoEMamba network) algorithm, a neural network is designed to predict the weight of TEADWA, and the weight adjustment of TEADWA is taken as the action space of reinforcement learning, so that the unmanned aerial vehicle system can dynamically adjust the behavior of TEADWA according to the environment state, rather than directly generating a motion command, which not only obtains the adaptability of deep learning, but also maintains the safety guarantee of TEADWA, and can better handle dynamic obstacles / other aircraft than using any method alone, and has better performance than traditional DWA and pure deep learning method, and increases the adaptability of the system to environmental changes, especially when the system faces birds, other unmanned aerial vehicles / aircraft or unpredictable moving obstacles, and makes safer and more efficient decisions.

[0581] Reference Figure 4 The embodiment of the application also provides an unmanned aerial vehicle patrol path decision device in a complex environment, which can implement the unmanned aerial vehicle patrol path decision method in a complex environment.

[0582] A graph construction unit is configured to construct an undirected topological graph with each to-be-patrolled position as a node.

[0583] A stationary distribution generation unit is configured to generate a stationary distribution of unmanned aerial vehicle patrol for each node according to the topological constraints and node importance of the undirected topological graph.

[0584] a transition matrix generating unit configured to generate a plurality of candidate state transition matrices with the steady state distribution, wherein each of the candidate state transition matrices comprises transition probabilities of the UAV between each of the nodes;

[0585] an environment evaluating unit configured to perform dynamic environment evaluation on obstacles of a path corresponding to each of the candidate state transition matrices, so as to determine a selectable speed and direction of the UAV on the selected path;

[0586] a command generating unit configured to dynamically adjust weights of the decision network according to the dynamic environment evaluation result, and to generate an execution motion command by using the decision network with the adjusted weights;

[0587] a movement control unit configured to control movement of the UAV between each of the to-be-patrolled positions according to the execution motion command.

[0588] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiments, the present device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0589] Embodiments of the present application further provide an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the embodiments of the present application when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer, a vehicle-mounted computer, etc.

[0590] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiments, the present device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0591] Please refer to Figure 5 , Figure 5 a hardware structure of an electronic device of another embodiment is shown, which comprises:

[0592] The processor 501 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0593] The memory 502 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 502 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 502 and are called and executed by the processor 501 to perform the method of the embodiments of the present application;

[0594] The input / output interface 503 is configured to realize information input and output.

[0595] The communication interface 504 is configured to realize the communication interaction between the device and other devices, and the communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0596] The bus 505 is configured to transmit information between various components (for example, the processor 501, the memory 502, the input / output interface 503, and the communication interface 504) of the device.

[0597] The processor 501, the memory 502, the input / output interface 503, and the communication interface 504 are connected to each other through the bus 505 to realize the communication connection between the device.

[0598] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the method of the present application.

[0599] It can be understood that the contents in the above method embodiments are applicable to the present storage medium embodiments, the functions specifically realized by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved by the present storage medium embodiments are also the same as those of the above method embodiments.

[0600] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0601] The embodiments described in the specification are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0602] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0603] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0604] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0605] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0606] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0607] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0608] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0609] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0610] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0611] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for unmanned aerial vehicle (UAV) patrol path decision in a complex environment, characterized in that, The method comprises the following steps: A non-directional topological graph is constructed with each to-be-patrolled position as a node; A steady-state distribution of the UAV patrolling each of the nodes is generated according to a topological constraint and a node importance of the non-directional topological graph; A plurality of candidate state transition matrices with the steady-state distribution are generated; each of the candidate state transition matrices comprises a transition probability of the UAV between each of the nodes; An obstacle on a path corresponding to each of the candidate state transition matrices is dynamically environmentally evaluated, so as to determine a selectable speed and direction of the UAV on the selected path; The weights of the decision network are dynamically adjusted according to the result of the dynamic environmental evaluation, and then an execution motion command is generated by using the decision network with the adjusted weights; The UAV is controlled to move between each of the to-be-patrolled positions according to the execution motion command; The weights of the decision network are dynamically adjusted according to the result of the dynamic environmental evaluation, and the method comprises the following steps: The weights of the decision network are dynamically adjusted according to the system state of the path corresponding to the target state transition matrix, and the method comprises the following steps: A system state, an output action and a reward function required by the adjusted decision network Actor-Critic are designed; A dynamic environment adaptability function is added to a traditional DWA to obtain a time-varying environment adaptive dynamic window method; The decision network Actor-Critic and the time-varying environment adaptive dynamic window method are combined to obtain an AC-time-varying environment adaptive dynamic window method, a state value estimation V(s) is output by a Critic network in the decision network, and the weights of an Actor network in the decision network are adjusted by using an advantage function, that is, the weight parameters in the time-varying environment adaptive dynamic window method; The Actor-Critic method replaces a recurrent neural network in a traditional Actor-Critic method with a MoEMamba network, the MoEMamba is a network fusing and mixing an expert network and a Mamba network, the two kinds of networks are alternately used, one layer of the expert network and one layer of the Mamba network are alternately used, and the alternation is repeated for a total of k layers; System state s t All information contained in the state of the UAV at time t is described, which is the basis of the AC-TEADWA method and provides the complete information required for the neural network to predict the AC-TEADWA weight, and the expression of the system state is: ； wherein, : system state at time t, containing the complete state information of the UAV at the tth moment; : the distance measurement value of the UAV to the obstacle at the current time t in the current UAV coordinate system; : distance measurement value of the previous control cycle k : distance measurement value of the previous control cycle : distance measurement value at a control cycle before the current UAV coordinate system l : distance measurement value at a control cycle before the current UAV coordinate system : distance measurement value of the previous control cycle g : distance measurement value of the previous control cycle : distance from current position of the drone to the target; : angle of the current position of the UAV to the target; : current translational speed of the drone; : current rotation speed of the drone; : obstacle distance change between consecutive time steps, capturing dynamic information; : time derivative of the drone-to-target distance, indicating whether the target is being approached; : current acceleration state vector of the UAV , is the linear acceleration of the UAV at the current time , is the angular acceleration of the UAV at the current time , : environmental complexity index, calculated by the entropy value of obstacle distribution; Specific forms of distance measurements: ; denotes a past time step, refers to the formula k , l , g ; Superscript denotes the measurement time; subscript coordinate system representing the current drone position; represents the number of measurement points of the laser radar, and is the distance between the position of the unmanned aerial vehicle and the nearest obstacle in the set direction. The system state is input into the decision network, and an action output by the decision network is obtained, and the action is expressed as: ； wherein, : heading item weight; : gap term weight; : speed term weight; : curvature distance component, additional weight of the new extended cost function; : dynamic environment adaptivity function component, additional weight of newly expanded cost function; The UAV receives an observation or a state of the environment, and outputs an action a, the action is a cost function weight of a next control period, and the weights will be used to generate a motion command of the UAV; The reward function is designed as follows: A large positive reward R1 is given when the UAV successfully arrives at a target position; a negative reward R2 is given when the UAV stays at the same position for a plurality of control periods, which is similar to the UAV being stuck; a large penalty, a negative R3, is given when the UAV collides with an obstacle, or other UAVs, or birds during movement; a negative reward R4 is given when the UAV is collided by a moving obstacle or other UAVs when being static; and the learning algorithm is guided to learn a safe and effective navigation strategy through the design of the reward function.

2. The method of claim 1, wherein, The steady-state distribution of the UAV patrolling each of the nodes is generated according to the topological constraint and the node importance of the non-directional topological graph, and the method comprises the following steps: The steady-state distribution is generated according to a topological constraint of the undirected topological graph, so that the probability of the unmanned aerial vehicle transferring to a target node is improved; therefore, a high-value target node and a node with a short attack time need to obtain more patrol resources, and the probability of the patrol unmanned aerial vehicle accessing these nodes is higher; An expression of the steady-state distribution is: ； ； wherein, , denotes the i-th node, V denotes a set of the nodes, denotes a target node a value of, denotes an attack time, denotes an importance weight parameter of the node denotes an importance weight parameter of the node denotes an importance weight parameter of the node n is a total number of nodes, denotes a steady state distribution of the target node a target steady state distribution denotes a probability distribution of the patroller at each target node after a long-term operation of the Markov chain.

3. The method of claim 1, wherein, The generation of the plurality of candidate state transition matrices with the steady-state distribution comprises the following steps: The plurality of candidate state transition matrices with the steady-state distribution and the target network topological structure are generated according to a multivariate random adaptive Markov steady-state matrix algorithm, comprising the following steps: First stage: hierarchical random matrix generation; generating a first stage random matrix satisfying topological constraints of the undirected topological graph, wherein self-loop probabilities are assigned using four distribution bins re-generating remaining non-self-loop probabilities ; Second stage: relative error iterative adjustment algorithm; The first-stage random matrix is adjusted according to an error between a probability distribution of the first-stage random matrix and the steady-state distribution, and specifically, a second-stage random matrix is obtained through relative error calculation, error proportion adjustment, row normalization and random periodic disturbance; Third stage: similarity value elimination algorithm; The second-stage random matrix in which a similarity value is greater than a preset similarity threshold is eliminated, and specifically, a third-stage random matrix is obtained through similarity value detection, random adjustment and row normalization to ensure that there is no too close value in the matrix, to enhance randomness and create a more natural random distribution; Fourth stage: matrix diversity guarantee algorithm; adjusting the third stage random matrices according to the mean absolute error between the probability distributions of the third stage random matrices, specifically by matrix difference degree calculation, diversity determination, large-scale random disturbance and re-adjustment of steady-state distribution to ensure the generation of a plurality of Markov state transition matrices with sufficient diversity, rather than variants of similar matrices, ultimately obtaining fourth stage random matrices as the candidate state transition matrices.

4. The method of claim 1, wherein, The dynamic environment assessment of the obstacles of the path corresponding to each candidate state transition matrix comprises the following steps: The dynamic environment assessment of the obstacles of the path corresponding to each candidate state transition matrix is performed according to a dynamic environment adaptability function, and a risk weighted sum of each obstacle is calculated to quantify the environment adaptability; An expression of the dynamic environment adaptability function (DEAF) is: ； wherein, : a dynamic environment adaptability function representing an environment adaptability evaluation value of the UAV under a given speed instruction; : linear velocity command of the UAV, controls the size of the forward or backward speed of the UAV; : angular velocity command of the drone, controls the magnitude and direction of the turn of the drone; : current system state estimated using weighted observation data, an environment representation containing obstacle position and velocity, where is the observation data of the i-th time instant of the UAV, observation weight is computed by the following expression: ； : observed data weights of the observed data, quantifying the importance of the set of observations in the state estimation; are memory weight coefficients for autocorrelation, time proximity and bias, starting from initial empirical values, periodically updated using an online adaptive method according to environmental properties settings; : observed autocorrelation score reflecting temporal correlation of environmental dynamics; : a set of parameters controlling the calculation of the autocorrelation scores; : observed time closeness score, emphasizing the importance of recent data; : observed bias score, capturing unusual changes in the environment; : bias sensitivity parameter, controlling the degree of attention of the algorithm to the bias; : number of dynamic obstacles currently perceived, including other drones; : based on the state estimation The calculated obstacle i presence probability, the value range is [0, 1], the greater the value indicates the higher the collision risk, measures the probability of obstacle i intersecting with the path of the unmanned aerial vehicle when the unmanned aerial vehicle executes the speed command The calculated obstacle i presence probability, the value range is [0, 1], the greater the value indicates the higher the collision risk, measures the probability of obstacle i intersecting with the path of the unmanned aerial vehicle when the unmanned aerial vehicle executes the speed command : based on the speed command of the UAV and the relative speed vector of the UAV and the obstacle i calculated based on the obstacle speed information in the state estimation Ω, measuring the speed and direction of the relative motion of the UAV and the obstacle; : speed of the drone the predicted minimum distance of the drone at time t to the obstacle i, the smaller the value the higher the potential collision risk; : Normalization constant, ensuring the range of the score term is appropriate, determined based on environmental characteristics and UAV dynamics parameters experiments; : A positive number, used as a safety parameter to prevent division by zero.

5. The method of claim 1, wherein, The steps of implementing the time-varying environment adaptive dynamic window method comprise the following steps: The dynamic environment adaptability function is added to the traditional DWA basis to obtain the time-varying environment adaptive dynamic window method; A target function of the time-varying environment adaptive dynamic window method is defined as: ； wherein, : linear velocity command of the UAV, controlling the speed of the UAV advancing or retreating; : angular velocity command of the drone, controls the magnitude and direction of the turn of the drone; : a heading component representing the angle difference between the advancing direction of the UAV and the target, measuring the degree of the UAV towards the target; : gap component, representing the distance of the UAV to the nearest obstacle on the current considered curvature, measuring the safety of the motion; : velocity component, representing the translational velocity of the drone, measuring the efficiency of the motion; : curvature distance component, representing the closest distance of the target point to the current considered curvature, measuring the relevance of the path to the target; : a dynamic environment adaptability function component representing the current speed command The degree of adaptability to the predicted environmental dynamic changes measures whether the UAV can effectively respond to the expected motion of dynamic obstacles in the environment while executing the current speed command; : weight corresponding to each component; Curvature distance component represents the closest distance of the target point to the current considered curvature, measures the relevance of the path to the target, and is expressed as follows: ； : distance attenuation factor >= 1, controls the degree of non-linearity of the distance influence; : proximity gain factor 0-1, increasing reward when target is very close to curvature; : proximity sensitivity, controls how steep the proximity reward is; : direction penalty factor, considering deviation of curvature line direction from target direction; : tangent direction of the curvature line; : direction of the target point relative to the drone; : radius of the drone, used to normalize distance values; : the shortest distance between the curvature of motion generated by the current speed combination (v, w) and the target point is the truncation parameter; : maximum distance threshold for normalization and limiting the range of distance influence.

6. The method of claim 1, wherein, The steps of implementing the AC-time-varying environment adaptive dynamic window method comprise the following steps: Historical observation data are acquired and preprocessed, and historical observations are converted to a current unmanned aerial vehicle coordinate system, and historical observations at different times are aligned to the same reference coordinate system of the current unmanned aerial vehicle, and a coordinate system conversion formula is defined as follows: wherein : current time; : general time offset parameter, including k, l or g, representing a time difference; : converted distance measurement, representing history observations at previous time instants in the coordinate system of the current time instant : at the moment, distance measurement values with the moment, the UAV coordinate system as a reference : from time to time, indicating the relative motion speed; : a set of state estimates of dynamic objects detected in the environment; : mixing coefficients, dynamically adjusted over time and observation quality , controlling the weight of the two conversion methods; : time derivative of coordinate transformation, indicates relative velocity of motion; : an evaluation function for calculating the mixing coefficient η; : Swish activation function; : a function to convert the polar form distance scan data into Cartesian coordinate points; : a forward projection function based on velocity prediction, considering trajectories of moving obstacle candidates including other aircraft; : function to convert points back to polar form from distance scan form; The neural network structure in the AC-time-varying environment adaptive dynamic window method is defined as: 1D four-channel input; The feature extraction layer processes spatially related scanning data; The convolution layer uses a Swish activation function; Feature fusion flattens the convolution output into a vector, and connects the target distance, angle and current speed; In the Actor-Critic architecture, the Actor and Critic networks share the same basic structure, except for the last layer; The training and output decision of the AC-time-varying environment adaptive dynamic window method are defined as: Three-stage training of different difficulty levels; The Actor network inputs system state information, including historical observations, target position, current speed, and outputs actions, including five TEADWA weight values: heading component weight , gap component weight , speed component weight , curvature distance component weight , dynamic environment adaptability function component weight ; The Critic network outputs a state value estimate V(s), and the advantage function is used to guide the improvement of the Actor.

7. An unmanned aerial vehicle patrol path decision device in a complex environment, characterized in that, The device is used to realize the method for unmanned aerial vehicle patrol path decision in complex environment as claimed in claim 1, and the device comprises: a graph construction unit configured to construct an undirected topological graph with each to-be-patrolled position as a node; a steady-state distribution generation unit configured to generate a steady-state distribution of unmanned aerial vehicle patrol for each node according to a topological constraint and a node importance of the undirected topological graph; a transition matrix generation unit configured to generate a plurality of candidate state transition matrices with the steady-state distribution; each candidate state transition matrix comprises a transition probability of the unmanned aerial vehicle between each node; an environment evaluation unit configured to perform dynamic environment evaluation on an obstacle of a path corresponding to each candidate state transition matrix, so as to determine a selectable speed and direction of the unmanned aerial vehicle on the selected path; a command generation unit configured to dynamically adjust a weight of a decision network according to a dynamic environment evaluation result, and then generate an execution motion command by using the decision network with the adjusted weight; a movement control unit configured to control the unmanned aerial vehicle to move between each to-be-patrolled position according to the execution motion command.

8. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor realizes the method as claimed in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to realize the method as claimed in any one of claims 1 to 6.

Images