An unmanned aerial vehicle dynamic obstacle avoidance method based on improved reinforcement learning

Abstract

The present application relates to the technical field of unmanned aerial vehicle flight control, and particularly relates to a dynamic obstacle avoidance method for unmanned aerial vehicle based on improved reinforcement learning, which obtains flight parameters of the unmanned aerial vehicle, obstacle distribution, tension of a flexible connecting member and release length of the flexible connecting member; calculates relative distance from the connecting member to the obstacle and tension change rate, and constructs an initial state vector in combination with the flight parameters; uses a self-attention network to fuse a historical state output time series enhanced feature vector; inputs a reinforcement learning strategy network to obtain an initial action vector containing speed, heading and release and control amount; when there is an out-of-limit risk in the tension or the relative distance, performs safety correction on the initial action to generate a final cooperative instruction and issue a bottom layer execution. The present application breaks the traditional obstacle avoidance limitation of only focusing on the body, takes into account the spatial form and stress state of the cable, effectively reduces the scratching, winding or traction disturbance, and improves the overall obstacle avoidance safety of the cable-equipped unmanned aerial vehicle system.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) flight control technology, and in particular to a dynamic obstacle avoidance method for UAVs based on improved reinforcement learning. Background Technology

[0002] As the operational scenarios for drones continue to expand, when performing tasks such as long-endurance power supply, media delivery, or tethered communication, drones typically need to be connected to ground equipment via flexible connectors such as cables and hoses. In complex low-altitude operating environments, tethered drones inevitably need to perform dynamic obstacle avoidance maneuvers during movement to avoid spatial obstacles such as building protrusions, trees, or equipment supports while maintaining normal operation.

[0003] Existing dynamic obstacle avoidance methods for drones typically rely primarily on the spatial distance between the drone and obstacles for obstacle avoidance decisions, paying less attention to the real-time status of the towed flexible connectors. Because the flexible connectors undergo spatial shape changes during flight depending on their trajectory and release length, control strategies based solely on drone collision avoidance are prone to situations where the drone has avoided the obstacle, but the cable still scrapes or becomes entangled with it. Furthermore, during obstacle avoidance maneuvers, the cable can suddenly taut, causing traction disturbances to the drone's flight attitude and reducing the overall safety of the system. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a dynamic obstacle avoidance method for UAVs based on improved reinforcement learning. It aims to improve upon the fact that existing UAV dynamic obstacle avoidance methods typically make obstacle avoidance decisions mainly based on the spatial distance between the UAV body and the obstacle, while paying less attention to the real-time status of the flexible connecting parts towed below.

[0005] This invention provides the following technical solution: a dynamic obstacle avoidance method for unmanned aerial vehicles (UAVs) based on improved reinforcement learning, comprising the following steps: S1. Obtain the current flight parameters of the UAV, obstacle distribution parameters, current tension of the flexible connector, tension at the previous moment, and current release length of the retraction device; S2. Calculate the relative distance between the flexible connector and the obstacle, and calculate the tension change rate based on the tension at the previous moment and the current tension. Then, concatenate the relative distance, the tension change rate, the current tension, and the current flight parameters to construct an initial state vector. S3. Obtain the pre-saved historical state vector, input the initial state vector and the historical state vector into the attention network model, and output the temporal enhancement feature vector. S4. Input the time-series enhanced feature vector into the reinforcement learning policy network, and output an initial action vector containing flight speed adjustment, heading adjustment and retraction device control. S5. When there is a risk of exceeding the limit in the current tension or relative distance in the initial state vector, perform a safety correction on the flight speed adjustment and the heading adjustment in the initial action vector, and combine the correction result with the control quantity of the retraction device to generate the final cooperative command. S6. The final collaborative command is transmitted to the flight controller of the UAV and the launch and take-off device for execution.

[0006] Preferably, in step S2, calculating the relative distance between the flexible connector and the obstacle specifically includes the following steps: Based on the current release length, the current tension, and the current position of the UAV and the preset ground connection point position in the current flight parameters, the spatial coordinates of each discrete node along the length direction of the flexible connector are calculated using the catenary model, and a node coordinate sequence is generated. Nodes closest to the UAV end, nodes with the lowest verticality, and nodes closest to the ground end are extracted from the node coordinate sequence and used as representative nodes of the dangerous segment. Calculate the distance between each representative node of the dangerous segment and the obstacle boundary corresponding to the obstacle distribution parameters, and determine the minimum value of each distance as the relative distance.

[0007] Preferably, in step S2, constructing the initial state vector specifically includes the following steps: Obtain preset distance normalized reference values, tension normalized reference values, and flight parameter normalized reference values; The relative distance, the tension change rate, the current tension, and the current flight parameters are normalized using the corresponding normalized reference values ​​to generate corresponding normalized distance features, normalized tension change features, normalized current tension features, and normalized flight features. The normalized distance feature, the normalized tension change feature, the normalized current tension feature, and the normalized flight feature are combined in a preset sequence to generate an initial state vector.

[0008] Preferably, in step S3, the output temporal enhancement feature vector specifically includes the following steps: Arrange the historical state vector and the initial state vector in chronological order to construct a temporal state feature matrix; The time-series state feature matrix is ​​multiplied by a preset query weight matrix, key weight matrix, and value weight matrix, respectively, to generate the corresponding query matrix, key matrix, and value matrix. Calculate the inner product of the query matrix and the key matrix, and process the result of the inner product using a normalized exponential function to generate an attention weight matrix; Calculate the product of the attention weight matrix and the value matrix, and output the temporal enhancement feature vector.

[0009] Preferably, the step of calculating the inner product of the query matrix and the key matrix, and processing the result of the inner product using a normalized exponential function to generate the attention weight matrix specifically includes the following steps: The relative distance and the rate of change of tension are extracted from the state vectors at each time step of the time-series state feature matrix, and normalized respectively to eliminate physical dimensions. The difference between the value 1 and the normalized relative distance is taken as the distance risk sub-item, and the absolute value of the normalized tension change rate is taken as the tension risk sub-item. The distance risk item and the tension risk item are linearly weighted using preset weighting coefficients to obtain the risk coefficient at each time step, and a time-series bias matrix is ​​constructed based on the difference between the risk coefficients at adjacent time steps. The attention weight matrix is ​​generated by adding the inner product of the time-series bias matrix, the query matrix, and the key matrix, and then performing a normalized exponential operation on the added result.

[0010] Preferably, in step S4, the output of the initial motion vector, which includes flight speed adjustment, heading adjustment, and retraction / deployment device control, specifically includes the following steps: The temporal enhancement feature vector is input into the fully connected hidden layer of the reinforcement learning policy network and processed by the activation function to output the hidden layer action features; The hidden layer action features are input into the action output layer of the reinforcement learning policy network, which outputs a continuous normalized action vector. Obtain the preset flight control physical constraint range and the retraction control physical constraint range, and use the flight control physical constraint range and the retraction control physical constraint range to perform denormalization mapping on the normalized action vector, separate the flight speed adjustment amount, the heading adjustment amount and the retraction device control amount, and combine them to generate the initial action vector.

[0011] Preferably, in step S5, generating the final collaborative instruction specifically includes the following steps: When it is determined that the current tension is greater than a preset tension threshold or the relative distance is less than a preset distance threshold and there is a risk of exceeding the limit, a preset speed attenuation coefficient is obtained, and the flight speed adjustment amount is multiplied by the speed attenuation coefficient to generate a corrected speed amount; The offset compensation angle in the direction away from the obstacle is calculated based on the obstacle distribution parameters in the initial state vector, and the offset compensation angle is superimposed on the heading adjustment amount to generate the corrected heading vector; The corrected velocity, the corrected heading vector, and the control quantity of the launch and recovery device are concatenated to generate the final coordinated command.

[0012] Preferably, the step of calculating the offset compensation angle in the direction away from the obstacle based on the obstacle distribution parameters in the initial state vector specifically includes the following steps: Two candidate offset compensation angles are generated on both sides of the obstacle based on the obstacle distribution parameters, and then combined with the flight speed adjustment, the heading adjustment and the retraction device control to obtain two candidate motion vectors. The candidate position of the UAV and the candidate release length of the flexible connector for the next control cycle are calculated based on each candidate motion vector, and the candidate tension of the flexible connector and its candidate relative distance to the obstacle are calculated using the catenary model. Candidate risk values ​​are calculated based on the deviation of the candidate tension from the preset tension threshold and the deviation of the candidate relative distance from the preset distance threshold, respectively. The candidate offset compensation angle corresponding to the candidate risk value with the smaller value is determined as the offset compensation angle.

[0013] Preferably, step S6 specifically includes the following steps: The final collaborative command is parsed to separate the UAV pose control data packet and the retractor drive data packet; Using a preset communication protocol, the UAV pose control data packet is converted into a flight control electrical signal, and the retraction device drive data packet is converted into a motor control electrical signal; The flight control electrical signals are transmitted to the flight controller via a communication bus, and the motor control electrical signals are transmitted to the main control board of the take-up and take-down device.

[0014] The present invention has the following beneficial effects: 1. In this invention, the tension state of the flexible connector and its relative distance from the obstacle are deeply integrated with the flight parameters of the UAV, breaking the limitation of traditional dynamic obstacle avoidance methods that only focus on the safety of the aircraft body. While achieving safe detour of the aircraft body, it effectively reduces the situation of scraping, entanglement or excessive tension of the towed cable, and effectively improves the overall obstacle avoidance safety and operational reliability of the cabled UAV system.

[0015] 2. In this invention, a self-attention network is used to extract temporal features from historical and current state vectors containing multiple moments. The attention weights for different historical moments are dynamically allocated in combination with distance and tension risk indicators. This enables the control model to capture the potential trends of the drooping shape evolution and stress fluctuations of the flexible connector in advance, reducing the impact of data jumps caused by sensor noise and sudden airflow interference, and enhancing the foresight and robustness of obstacle avoidance decisions.

[0016] 3. In this invention, the future flight position and cable status under candidate obstacle avoidance commands are predicted and evaluated by using a catenary physical model. Based on the inference output of the reinforcement learning network, safety correction constraints that conform to the actual operating boundary are superimposed, which avoids the output of uncontrollable dangerous actions by pure data-driven algorithms in complex scenarios and ensures the smooth coordination of flight speed adjustment and the driving action of the retraction device. Attached Figure Description

[0017] Figure 1 This is a flowchart of a dynamic obstacle avoidance method for unmanned aerial vehicles (UAVs) based on improved reinforcement learning, as proposed in this invention. Detailed Implementation

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This invention provides a dynamic obstacle avoidance method for unmanned aerial vehicles (UAVs) based on improved reinforcement learning, such as... Figure 1 As shown, it includes the following steps: S1. Obtain the current flight parameters of the UAV, obstacle distribution parameters, current tension of the flexible connector, tension at the previous moment, and current release length of the retraction device; Specifically, for step S1, the system synchronously collects low-level data through multi-source sensors mounted on the UAV and the deployment / retraction device at the ground connection end. The UAV's current flight parameters are acquired in real time by the onboard flight controller in conjunction with the real-time dynamic differential positioning module and the inertial measurement unit. The current flight parameters are specifically composed of the UAV's three-dimensional spatial coordinates in the world coordinate system, its three-dimensional flight speed, and its heading angle in the airframe, which are used to characterize the current motion state of the UAV.

[0020] Obstacle distribution parameters are acquired using an onboard LiDAR fixed to the UAV. The LiDAR scans the surrounding environment at a fixed frequency to generate 3D point cloud data. The environmental perception module first performs voxel filtering downsampling on the point cloud data to remove interference from suspended impurities in the air. The grid size for voxel filtering can be set to 0.1 meters, and the maximum sensing distance threshold can be set to 20.0 meters. The set of 3D spatial coordinates of all valid obstacle points retained after filtering is output as the obstacle distribution parameters, used for calculating the distance between representative nodes of the danger zone and the obstacle boundary. Simultaneously, the 3D Euclidean distance from the geometric center of the UAV to each point is calculated from the aforementioned valid point cloud. The coordinates of the point with the smallest Euclidean distance and its corresponding relative azimuth angle under the UAV system are selected as near-end features in the obstacle distribution parameters, used for determining the risk of exceeding limits in the independent claims.

[0021] The current and previous tensions of the flexible connector are acquired using miniature tension sensors mounted on the guide wheel base of the deployment and take-off device. Considering that high-frequency aerodynamic vibrations of the UAV rotor will be transmitted to the sensor through the flexible connector, causing high-frequency noise in the measurement signal, the system employs a first-order hysteresis filtering algorithm to smooth the raw tension signal output by the tension sensor in the time domain. The specific filtering formula is as follows: ; in The tension at the current moment after processing, in Newtons; This is the raw tension measurement value collected by the tension sensor at the current moment, in Newtons; The tension value output in the previous control cycle, in Newtons; The filter coefficient is dimensionless, ranging from 0 to 1. All physical dimensions of the tension terms involved in the weighted calculation are uniformly set to Newtons, ensuring dimensional consistency in the weighted operation. In practical operating scenarios, the filter coefficient can be set to 0.2 to balance the sensitivity to sudden tension changes with signal smoothness.

[0022] The current release length of the deployment and take-off device is calculated in real time using an incremental rotary photoelectric encoder mounted on the shaft end of the deployment and take-off servo motor. The system uses the state where the flexible connector is fully retracted and the UAV is docked at the landing pad as the zero point of length. The formula for calculating the current release length is: ; in This is the current release length, in meters. The effective winding radius of the winding device drum, in meters; This represents the cumulative number of valid pulses output by the encoder after quadrature decoding since system initialization. Release counts are positive, and recovery counts are negative. The nominal total number of pulses for one revolution of the encoder can be taken as 2048. The above formula applies to single-layer winding scenarios, in which case... As a fixed constant, a value of 0.15 meters can be taken; for multi-layer winding scenarios, the average value of the winding radius of each layer can be taken as the value. The equivalent approximation is .

[0023] This step completes the multi-source synchronous acquisition and digital quantization of the UAV's physical state, the spatial distribution of surrounding obstacles, and the tension and release length of the flexible connectors. It eliminates the data blind spots that rely on a single sensor and provides complete underlying data input for the subsequent steps to construct the initial state vector.

[0024] S2. Calculate the relative distance between the flexible connector and the obstacle, and calculate the rate of change of tension based on the tension at the previous moment and the current tension. Then, combine the relative distance, the rate of change of tension, the current tension, and the current flight parameters to construct the initial state vector. Furthermore, in S2, calculating the relative distance between the flexible connector and the obstacle specifically includes the following steps: Based on the current release length, current tension, and current flight parameters, the current position of the UAV and the preset ground connection point position are used to calculate the spatial coordinates of each discrete node along the length direction of the flexible connector using the catenary model, and generate a node coordinate sequence. Extract the nodes closest to the UAV end, the node with the lowest verticality, and the node closest to the ground end from the node coordinate sequence as representative nodes of the dangerous segment; Calculate the distance between the representative node of each dangerous segment and the obstacle boundary corresponding to the obstacle distribution parameters, and determine the minimum value of each distance as the relative distance.

[0025] Furthermore, in S2, constructing the initial state vector specifically includes the following steps: Obtain preset distance normalized reference values, tension normalized reference values, and flight parameter normalized reference values; Normalization is performed on the relative distance, tension change rate, current tension, and current flight parameters using the corresponding normalized reference values ​​to generate the corresponding normalized distance features, normalized tension change features, normalized current tension features, and normalized flight features. The normalized distance feature, normalized tension change feature, normalized current tension feature, and normalized flight feature are combined in a preset sequence to generate an initial state vector.

[0026] Specifically, after completing the multi-source data acquisition in step S1, step S2 first uses a catenary model to geometrically model the spatial morphology of the flexible connector. Using the current three-dimensional position coordinates of the UAV obtained in S1 and the pre-calibrated ground connection point coordinates as the boundary conditions at both ends, a local two-dimensional coordinate system is established in a vertical plane containing these two points. Let the UAV position coordinates be... The coordinates of the ground connection point are The formulas for calculating the horizontal distance and vertical height difference between the two endpoints are as follows: ; ; in The distance is horizontal, in meters. The vertical height difference is expressed in meters. Within the local coordinate system, with the lowest point of the catenary as the origin, the vertical height of the flexible connector satisfies the standard catenary equation: ; in Horizontal position within the local coordinate system The vertical height of the node, in meters; These are characteristic parameters of the catenary, in meters; The horizontal tension component of the flexible connector is expressed in Newtons. This refers to the weight per unit length of the flexible connector, expressed in Newtons per meter. It is pre-calibrated based on the connector material specifications. For a power cable with a diameter of 4 mm, the following value can be used: Newtons per meter. Let the horizontal coordinates of the ground connection point and the UAV connection point in the local coordinate system be respectively... and ,satisfy Then the equations for the arc length constraint and the elevation difference constraint are as follows: ; ; in The current release length, in meters, is calculated by the encoder of the take-up and release device in S1. The catenary characteristic parameters are solved using Newton's iteration method for the above nonlinear equations. and local horizontal coordinates at the ground end The iterative convergence threshold is set to The maximum number of iterations is set to 50.

[0027] After obtaining the catenary parameters, the flexible connector is divided into equal intervals along the arc length direction. Each node can be selected. . No. The arc length coordinates of each node are ,in Take 0 to Integers, 0 corresponds to the ground end. For the drone side, the two-dimensional coordinates of each node in the local coordinate system are obtained by the parametric relationship between the arc length and the catenary equation. Then, the coordinates are transformed to the world coordinate system by the coordinate rotation matrix to generate a node coordinate sequence, where each node is a three-dimensional coordinate in the world coordinate system, with the unit being meters.

[0028] Extract three representative nodes from the node coordinate sequence, and take the first one. Each node is selected as a representative node closer to the UAV. The node with the smallest altitude coordinate value in the world coordinate system is selected as the representative node with the lowest verticality. The 0th node is selected as the representative node closer to the ground. The three-dimensional Euclidean distances from the above three representative nodes to each point in the effective obstacle point cloud coordinate set output in S1 are calculated respectively. The minimum value among all distance calculation results is taken as the relative distance. ; in For the first Each dangerous segment represents the three-dimensional coordinates of a node, in meters; The first point cloud of obstacles The three-dimensional coordinates of each valid point are given in meters. The relative distance between the flexible connector and the obstacle during the current control cycle is expressed in meters.

[0029] After completing the relative distance calculation, use the current tension output in S1. tension with the previous moment Calculate the rate of change of tension: ; in This is the rate of change of tension, measured in Newtons per second. The control period, in seconds, is consistent with the sampling period in step S1 and can be taken as follows: Second.

[0030] After calculating the physical quantities, normalization is performed on each quantity to eliminate dimensional differences. A preset distance normalization reference value is obtained. tension normalized reference value Normalized reference value for rate of change of tension and including spatial location boundaries Maximum flight speed The normalized reference values ​​for flight parameters are as follows. The normalization formulas for each parameter are: ; in , , These are the normalized distance, current tension, and tension change characteristics, respectively. and These represent the current three-dimensional spatial coordinate components and three-dimensional flight velocity components of the UAV in S1, respectively, in meters and meters per second. Corresponding to the forward, horizontal, and vertical directions of the world coordinate system; The heading angle obtained in S1 is in radians; , , These are the corresponding normalized flight characteristics. Each reference value can be taken separately. rice, Newton Newton per second rice, meters per second.

[0031] Will , , And triaxial Triaxial And single-dimensional By splicing together a fixed sequence from environmental features to its own motion features, a one-dimensional initial state vector of length 10 is generated.

[0032] This step transforms the nonlinear physical form of the flexible connector in three-dimensional space into precise distance values, and constructs a standard feature combination through dimensionless splicing, providing a structured environmental state representation containing machine-line cooperative physical information for the real-time inference of the subsequent policy network.

[0033] S3. Obtain the pre-saved historical state vector, input the initial state vector and historical state vector into the attention network model, and output the temporal enhancement feature vector. Furthermore, in S3, the output of the temporal enhancement feature vector specifically includes the following steps: Arrange the historical state vector and the initial state vector in chronological order to construct a temporal state feature matrix; The time-series state feature matrix is ​​multiplied by the preset query weight matrix, key weight matrix, and value weight matrix respectively, and the corresponding query matrix, key matrix, and value matrix are generated by mapping. Calculate the inner product of the query matrix and the key matrix, and process the result of the inner product using a normalized exponential function to generate the attention weight matrix; Calculate the product of the attention weight matrix and the value matrix, and output the temporal enhancement feature vector.

[0034] Further, the inner product of the query matrix and the key matrix is ​​calculated, and the result of the inner product is processed using a normalized exponential function to generate the attention weight matrix. This process specifically includes the following steps: The relative distance and tension change rate are extracted from the state vectors at each time step of the temporal state feature matrix, and then normalized to eliminate physical dimensions. The difference between the value 1 and the normalized relative distance is taken as the distance risk sub-item, and the absolute value of the normalized tension change rate is taken as the tension risk sub-item. The distance risk sub-item and the tension risk sub-item are linearly weighted using preset weighting coefficients to obtain the risk coefficient at each time step, and a time-series bias matrix is ​​constructed based on the difference in risk coefficients between adjacent time steps. The attention weight matrix is ​​generated by adding the inner product of the time-series bias matrix, the query matrix, and the key matrix, and then performing a normalized exponential operation on the added result.

[0035] Specifically, after generating the initial state vector for the current control cycle in step S2, step S3 reads the aforementioned initial state vector and the pre-saved historical state vector from the memory. The system is set to a length of... The time window, in this embodiment, is taken as... and maintain the previous A queue of historical state vectors for each control cycle. When the system runs for the first time, the first initial state vector is copied. The queue is filled in turn; after feature extraction for each control cycle is completed, the current initial state vector is written to the tail of the queue, and the state vector at the head of the queue is deleted. Let the current control cycle be... Arranged in chronological order Construct a time-series state feature matrix using the historical state vectors and the current initial state vectors for each control cycle: ; in, The current initial state vector generated in step S2, to These are pre-saved historical state vectors. Each state vector has a length of 10, and the temporal state feature matrix is... The dimension is .

[0036] The system loads a query weight matrix that is jointly trained with the reinforcement learning policy network and pre-saved. Key weight matrix Sum weight matrix The weight matrices are kept fixed during online obstacle avoidance. All three weight matrices have a dimension of [missing value]. , where the feature mapping dimension A value of 16 can be chosen. Multiply the time-series state feature matrix by the three weight matrices respectively to obtain the query matrix, key matrix, and value matrix: ; ; ; in, For querying the matrix, The key matrix, The matrix is ​​a value matrix, and the dimensions of all three are 1. Since the initial state vector output in step S2 is a dimensionless vector, and the three pre-trained weight matrices also use dimensionless values, the resulting query matrix, key matrix, and value matrix do not have physical dimensions.

[0037] Normalized distance features and normalized tension change features are extracted from the state vectors at each time step of the time-series state feature matrix. To prevent sensor data from exceeding the normalization reference range set in step S2, the extracted features are subjected to amplitude limiting processing. ; ; in, This is the time index within the time window, with values ​​ranging from 1 to... ; For the first Normalized distance features at each time step; For the first Normalized tension variation characteristics at each moment; and These are the dimensionless distance characteristics and dimensionless tension change characteristics after amplitude limiting; This means that the input value is restricted to between 0 and 1.

[0038] The difference between the value 1 and the normalized distance feature after limiting is taken as the distance risk sub-item, and the normalized tension change feature after limiting is taken as the tension risk sub-item: ; ; in, For the first The distance risk sub-item at any given moment increases as the relative distance between the flexible connectors decreases; For the first The tension risk sub-item at each time point increases with the absolute value of the tension change rate; both are dimensionless. Using preset dimensionless weighting coefficients, a linear weighting of the above two items is applied to obtain the risk coefficient for each time point: ; in, For the first The dimensionless risk coefficient at any given moment; This is the distance risk weighting coefficient; For the tension risk weighting coefficient, satisfying In this embodiment Take 0.6, We set it to 0.4. This step uses dimensionless processing to ensure dimensional consistency when distance risk and tension risk are linearly added.

[0039] Calculate the change in risk based on the risk coefficients at adjacent time points: ; in, For the first The change in risk at each moment relative to the previous moment. Take 2 to Based on the cumulative relationship of risk changes at adjacent time points along the time axis, a time-series bias matrix is ​​constructed. Define the query time Key Moments The risk volatility difference between them is the corresponding element of the time series bias matrix: ; in, Timing bias matrix The Line 1 Column elements; and Take 1 to ; This is a dimensionless bias gain coefficient, which is set to 1.0 in this embodiment. When the risk at the key moment is higher than that at the query moment, the elements of this matrix are positive, so that the system assigns a positive bias to the high-risk moment when calculating attention.

[0040] Calculate the scaled inner product of the query matrix and the transpose of the key matrix to obtain the basic attention score matrix: ; in, For dimension The basic attention score matrix; This is the transpose of the key matrix; To limit the anomalous growth of the inner product value, a dimensionless scaling factor is used. The dimensionless temporal bias matrix is ​​added element-wise to the base attention score matrix to generate the risk-biased attention score matrix. ; in, This is the risk-biased attention score matrix. Normalized exponentiation is performed on each row of this matrix to generate the attention weight matrix: ; in, Attention weight matrix The Line 1 Column elements; For the corresponding elements of the risk-biased attention score matrix; For summation index; It is a natural exponential function. The sum of the elements in each row of the matrix is ​​1.

[0041] Calculate the product of the attention weight matrix and the value matrix to obtain the output feature matrix: ; in, For dimension The output feature matrix. Extract the first feature from the output feature matrix corresponding to the current control cycle. Rows as temporal enhancement feature vectors: ; in, The time-enhanced feature vector output in the current control cycle has a length of That is, 16; For the current moment, the first Attention weights for each time-state; The first value matrix Row feature vectors.

[0042] This step enables the output features to include temporal information about the flexible connector's approach to obstacles and tension changes, providing input for the reinforcement learning policy network inference in step S4.

[0043] S4. Input the temporal enhancement feature vector into the reinforcement learning policy network, and output the initial action vector containing the flight speed adjustment, heading adjustment and retraction device control. Furthermore, in S4, the output of the initial motion vector, which includes flight speed adjustment, heading adjustment, and retraction / deployment control, specifically includes the following steps: The temporal augmentation feature vector is input into the fully connected hidden layer of the reinforcement learning policy network and processed by the activation function to output the hidden layer action features; The hidden layer action features are input into the action output layer of the reinforcement learning policy network, which outputs a continuous normalized action vector. Obtain the preset physical constraints of flight control and retraction control. Use the physical constraints of flight control and retraction control to perform inverse normalization mapping on the normalized motion vector, separate the flight speed adjustment, heading adjustment and retraction device control, and combine them to generate the initial motion vector.

[0044] Specifically, after obtaining the 16-length temporal augmentation feature vector output in step S3, step S4 uses it as the forward inference input to the reinforcement learning policy network. The network parameters of the reinforcement learning policy network are determined by pre-training and remain fixed during the online inference phase; it employs a multilayer perceptron architecture. The temporal augmentation feature vector is input into the fully connected hidden layer of the reinforcement learning policy network. In this embodiment, the fully connected hidden layer contains two layers, each with 64 neurons. The forward propagation calculation formula for the fully connected hidden layer is: ; ; in The temporal enhancement feature vector generated in step S3 has a length of 16; A fixed weight matrix with dimensions of 64 rows and 16 columns is pre-trained for the first hidden layer; A fixed weight matrix with dimensions of 64 rows and 64 columns is pre-trained for the second hidden layer; and The first and second hidden layers are pre-trained with fixed bias vectors, respectively. It is a linear rectification activation function; This is the output vector of the first hidden layer, with a length of 64; The hidden layer action features are output by the second hidden layer after nonlinear mapping. The length is 64, which contains high-dimensional abstract decision information of the drone's obstacle avoidance actions.

[0045] The hidden layer action features are input into the action output layer of the reinforcement learning policy network. The action output layer contains five neurons, corresponding to the three control components of the UAV's three-dimensional flight speed, the heading angle control component, and the deployment / retraction device control component, respectively. The forward propagation calculation formula for the action output layer is as follows: ; in A fixed weight matrix with a dimension of 5 rows and 64 columns is pre-trained for the action output layer; Pre-train fixed bias vectors for the action output layer; This is the hyperbolic tangent activation function, used to smoothly compress the output value of a neural network to within a fixed boundary; The continuous normalized motion vector output by the motion output layer has five dimensions whose values ​​are distributed between -1 and +1, and are dimensionless reference control values.

[0046] Obtain the preset physical constraints for flight control and deployment / retraction control. In this embodiment, the forward, lateral, and vertical flight speed adjustment components share the same maximum constraint boundary, which is set to 1.0 m / s; the maximum constraint boundary for heading adjustment is set to 0.5 radians / s; and the maximum constraint boundary for deployment / retraction control is set to 0.5 m / s. The normalized motion vector is then denormalized using the aforementioned physical constraints for flight control and deployment / retraction control. The calculation formula is as follows: ; ; ; in To extract normalized action vectors The dimensionless subvector formed by the first to third elements; and These are the fourth and fifth dimensionless scalar elements of the normalized action vector, respectively. This represents the maximum constraint boundary for flight speed adjustment. This represents the maximum constraint boundary for heading adjustment. This represents the maximum constraint boundary for the control quantity of the take-up and release device; The flight speed adjustment quantity in actual physical dimensions after inverse normalization is composed of three speed adjustment components: forward, lateral, and vertical, with the unit being meters per second. This is the heading adjustment in actual physical dimensions, measured in radians per second. This is a control quantity of the retraction and deployment device in actual physical dimensions. A positive value represents the drive motor releasing the flexible connector, and a negative value represents the drive motor retracting the flexible connector. The unit is meters per second.

[0047] The separated flight speed adjustment, heading adjustment, and launch / recapture control quantities are concatenated according to the data frame format to generate an initial motion vector containing five-dimensional physical action commands.

[0048] This step restores the dimensionless continuous motion features generated by the neural network into specific control quantities that conform to the physical boundary constraints of the UAV and the receiver / discharger, providing a reference command with definite physical meaning for subsequent safety corrections when risks exceed limits.

[0049] S5. When there is a risk of exceeding the limit in the current tension or relative distance in the initial state vector, perform safety correction on the flight speed adjustment and heading adjustment in the initial action vector, and combine the correction result with the control quantity of the launch and recovery device to generate the final coordinated command. Furthermore, in S5, generating the final cooperative instruction specifically includes the following steps: When it is determined that there is a risk of exceeding the limit due to the current tension being greater than the preset tension threshold or the relative distance being less than the preset distance threshold, the preset speed attenuation coefficient is obtained, and the flight speed adjustment amount is multiplied by the speed attenuation coefficient to generate the corrected speed amount; The offset compensation angle in the direction away from the obstacle is calculated based on the obstacle distribution parameters in the initial state vector, and the offset compensation angle is superimposed on the heading adjustment to generate the corrected heading vector; The corrected speed, corrected heading vector, and launch / recovery control parameters are concatenated to generate the final coordinated command.

[0050] Furthermore, calculating the offset compensation angle in the direction away from the obstacle based on the obstacle distribution parameters in the initial state vector specifically includes the following steps: Two candidate offset compensation angles are generated on both sides of the obstacle based on the obstacle distribution parameters, and then combined with the flight speed adjustment, heading adjustment and retraction device control values ​​to obtain two candidate motion vectors. The candidate position of the UAV and the candidate release length of the flexible connector for the next control cycle are calculated based on each candidate motion vector, and the candidate tension of the flexible connector and its candidate relative distance to the obstacle are calculated using the catenary model. Candidate risk values ​​are calculated based on the deviation of candidate tension from a preset tension threshold and the deviation of candidate relative distance from a preset distance threshold, respectively. The candidate offset compensation angle corresponding to the candidate risk value with the smaller value is determined as the offset compensation angle.

[0051] Specifically, after obtaining the initial motion vector output in step S4, step S5 extracts the current tension from the initial state vector constructed in step S2. The relative distance between the flexible connector and the obstacle And compare it with a preset threshold. The preset tension threshold is denoted as... The value can be set to 40.0 Newtons; the preset distance threshold is denoted as... 2.0 meters can be selected. When or When a risk of exceeding limits is identified, a safety correction is performed; when and When the initial action vector output in step S4 is used directly as the final collaborative instruction, the input vector is used directly as the final collaborative instruction.

[0052] When there is a risk of exceeding limits, the flight speed adjustment amount output in step S4 is first attenuated. A preset speed attenuation coefficient is then obtained. In this embodiment Take 0.5 and calculate the corrected velocity: ; in, To correct for the velocity, it consists of three velocity components: forward, lateral, and vertical, with the unit being meters per second; The flight speed adjustment amount output in step S4 is expressed in meters per second. This is a dimensionless velocity attenuation coefficient. In this calculation, the velocity adjustment is multiplied by the dimensionless coefficient, so there is no issue of dimension inconsistency.

[0053] Subsequently, two candidate offset compensation angles are generated based on the obstacle distribution parameters obtained in step S1. The environment perception module then uses the relative azimuth angle of the nearest obstacle point in the obstacle point cloud within the horizontal plane of the UAV system. First candidate offset compensation angles are generated on both sides away from the main body of the obstacle. Second candidate offset compensation angle Among them, relative azimuth angle The unit is radians, and the range of values ​​is... In this embodiment, the candidate offset compensation angle is superimposed on the heading adjustment amount, and its unit is the same as the heading adjustment amount, both being radians per second. radians per second Radius per second. Each candidate offset compensation angle is combined with the corrected velocity, the heading adjustment output in step S4, and the launch / retraction control quantity to obtain two candidate motion vectors.

[0054] Regarding the first Given a candidate action vector, first calculate the candidate heading adjustment: ; in, Choose 1 or 2, when The first candidate offset compensation angle corresponds to the time. ,when The corresponding second candidate offset compensation angle ; For the first The candidate heading adjustment amount corresponding to each candidate action vector, in radians per second; The heading adjustment amount output in step S4 is expressed in radians per second. For the first One candidate offset compensation angle, in radians per second.

[0055] The current flight speed of the drone obtained in step S1 The candidate flight speed is calculated together with the corrected velocity, and the candidate heading adjustment is applied to the horizontal velocity direction to obtain the candidate UAV position for the next control cycle. ; ; in, For the first The candidate flight speeds corresponding to each candidate action vector are expressed in meters per second. The rotation matrix is ​​about the vertical axis of the world coordinate system, and its input parameters are... For candidate heading change angles, the unit is radians; To control the cycle, the unit is seconds, and a value of 0.05 seconds can be used; Candidate locations for the drone in the next control cycle, in meters; The current three-dimensional position of the UAV obtained in step S1 is in meters; and Both are three-dimensional velocity vectors in the world coordinate system, with units of meters per second. The candidate offset compensation angle participates in the candidate position prediction through a rotation matrix, so that the two candidate motion vectors correspond to different orbital directions.

[0056] Based on the current release length obtained in step S1 and the control quantity of the take-up and extend device output in step S4 Calculate the candidate release length of the flexible connector: ; in, For the first The candidate release length corresponding to each candidate action vector, in meters; This is the current release length, in meters. This is the control value for the release and recovery device, measured in meters per second. A positive value indicates release, and a negative value indicates recovery.

[0057] Candidate positions Candidate release length Substituting the preset ground connection point locations and obstacle distribution parameters obtained in step S1 into the catenary model in step S2, the coordinate sequence of candidate flexible connector nodes is calculated. The relative distance between candidates is then calculated based on the candidate node coordinate sequence. , which is the minimum distance from the representative node of the dangerous segment of the candidate flexible connector to the boundary of the obstacle point cloud, in meters.

[0058] Simultaneously, candidate tensions at the UAV tip are calculated based on the horizontal tension components obtained from the catenary calculation, ensuring they have a consistent physical meaning with the tip tension threshold measured by the tension sensor. The formula for calculating candidate tension is: ; in, For the first The candidate tension at the end of the UAV corresponding to each candidate motion vector is expressed in Newtons. The horizontal tension component of the candidate catenary, expressed in Newtons, is obtained through iterative solutions to the candidate catenary equations, or based on the relational formula. Calculated; The unit length gravity of the flexible connector is used as preset in step S2, and the unit is Newtons per meter; These are the characteristic parameters of the candidate catenary, in meters; The horizontal coordinates of the candidate connection points of the UAV within the candidate local coordinate system are given in meters. It is a hyperbolic sine function. Because... Characterizing the end tension of the flexible connector, it can be compared with a preset tension threshold. Perform a consistency comparison.

[0059] Candidate risk values ​​are calculated based on the deviations of candidate tension from a preset tension threshold and the deviations of candidate relative distances from a preset distance threshold, respectively. To avoid inconsistencies in dimensions caused by directly adding tension and distance, a dimensionless risk value is constructed using normalized deviations. ; in, For the first The dimensionless candidate risk value of each candidate action vector; This refers to the tension risk weighting coefficient. , where is the distance risk weighting coefficient, and both are dimensionless numbers; in this embodiment, both are taken as 1.0. This is a function that maximizes the input. Both terms in the formula are dimensionless proportional values, allowing for linear weighting.

[0060] Compare two candidate risk values, and determine the candidate bias compensation angle corresponding to the candidate risk value with the smaller value as the bias compensation angle. The offset compensation angle is then superimposed on the heading adjustment output in step S4 to generate the corrected heading vector. ; in, To correct the heading vector, the unit is radians per second; The final selected offset compensation angle is expressed in radians per second.

[0061] Finally, the corrected velocity amount will be... Correcting the heading vector The control quantity of the take-up and extend device output in step S4 By concatenating the commands according to the preset instruction sequence, the final collaborative instruction is generated: ; in, For the final coordination instruction, the square brackets in the formula represent the concatenation of data frame formats of different control fields, not the addition operation between different physical quantities.

[0062] This step modifies the initial action based on tension and distance risks, so that the output command simultaneously considers the UAV's orbital direction, flight speed, and the retraction and extension status of the flexible connector, reducing the obstacle avoidance control risks when the flexible connector approaches an obstacle or experiences a sudden increase in tension.

[0063] S6. The final coordination command is transmitted to the UAV's flight controller and launch / recapture device for execution.

[0064] Furthermore, S6 specifically includes the following steps: The final coordinated command is parsed to separate the UAV pose control data packet and the retractor drive data packet. Using a preset communication protocol, the UAV pose control data packets are converted into flight control electrical signals, and the retraction device drive data packets are converted into motor control electrical signals; Flight control electrical signals are transmitted to the flight controller via a communication bus, and motor control electrical signals are transmitted to the main control board of the retraction and deployment device.

[0065] Specifically, after obtaining the final cooperative instruction generated in step S5, step S6 performs hardware-level data parsing and communication delivery of the instruction. The system unpacks the final cooperative instruction according to the preset data frame reading rules and separates it into two independent data branches in memory. The corrected velocity and corrected flight vectors are extracted and encapsulated into a UAV pose control data packet; the retractor control quantities are extracted and encapsulated separately into a retractor drive data packet.

[0066] After data packet separation, the system uses a preset communication protocol to convert the control data packets into control electrical signals recognizable by the underlying hardware. For the take-up / retractor drive data packets, the system converts the linear velocity-based control quantities of the take-up / retractor into the target motor speed. The conversion formula is as follows: ; in The target rotational speed of the drive motor for the take-up and release device, expressed in revolutions per minute; The control quantity for the take-up and release device generated in step S5 is expressed in meters per second. The effective winding radius of the take-up and unwinding device drum obtained in step S1 is in meters. Here, the constant of 0.15 meters under the single-layer winding scenario is used. Pi is a constant. After calculating the target rotational speed, the main control chip maps it into a motor control electrical signal in the form of voltage pulses through a pulse width modulation protocol or a digital-to-analog converter circuit.

[0067] For UAV attitude control data packets, the system performs discretization-quantization conversion on the continuous floating-point format correction velocity and correction head vector according to the micro-aircraft communication protocol, generating digital transmission electrical signals. The quantization calculation formula is as follows: ; ; in The converted three-dimensional velocity integer discrete data; The converted heading integer discrete data; The corrected velocity value generated in step S5, in meters per second; The corrected flight vector generated in step S5, in radians per second; The preset velocity quantization resolution is set to 0.01 meters per second. The preset navigation vectorization resolution is set to 0.01 radians per second. This is the rounding function. The system converts the above discrete integer data into flight control electrical signals containing a header, payload, and check bit, according to the communication data link layer standard.

[0068] After completing the signal protocol conversion, the system performs physical layer signal transmission through the airborne hardware interface. The encapsulated flight control electrical signals are sent to the UAV's flight controller via a universal asynchronous serial bus, which drives the multi-rotor's underlying attitude and dynamic response. At the same time, the motor control electrical signals are transmitted to the main control board of the retraction device via the controller's local area network bus, driving the servo motors to perform the corresponding retraction and extension physical displacement actions.

[0069] This step transforms the abstract numerical instructions jointly output by the neural network and the physical model into driving signals for the underlying actuators, thus completing the physical execution closed loop for collaborative dynamic obstacle avoidance between the UAV and the flexible connector.

[0070] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic obstacle avoidance method for unmanned aerial vehicles (UAVs) based on improved reinforcement learning, characterized in that, Includes the following steps: S1. Obtain the current flight parameters of the UAV, obstacle distribution parameters, current tension of the flexible connector, tension at the previous moment, and current release length of the retraction device; S2. Calculate the relative distance between the flexible connector and the obstacle, and calculate the tension change rate based on the tension at the previous moment and the current tension. Then, concatenate the relative distance, the tension change rate, the current tension, and the current flight parameters to construct an initial state vector. S3. Obtain the pre-saved historical state vector, input the initial state vector and the historical state vector into the attention network model, and output the temporal enhancement feature vector. S4. Input the time-series enhanced feature vector into the reinforcement learning policy network, and output an initial action vector containing flight speed adjustment, heading adjustment and retraction device control. S5. When there is a risk of exceeding the limit in the current tension or relative distance in the initial state vector, perform a safety correction on the flight speed adjustment and the heading adjustment in the initial action vector, and combine the correction result with the control quantity of the retraction device to generate the final cooperative command. S6. The final collaborative command is transmitted to the flight controller of the UAV and the launch and take-off device for execution.

2. The UAV dynamic obstacle avoidance method based on improved reinforcement learning according to claim 1, characterized in that, In step S2, calculating the relative distance between the flexible connector and the obstacle specifically includes the following steps: Based on the current release length, the current tension, and the current position of the UAV and the preset ground connection point position in the current flight parameters, the spatial coordinates of each discrete node along the length direction of the flexible connector are calculated using the catenary model, and a node coordinate sequence is generated. Nodes closest to the UAV end, nodes with the lowest verticality, and nodes closest to the ground end are extracted from the node coordinate sequence and used as representative nodes of the dangerous segment. Calculate the distance between each representative node of the dangerous segment and the obstacle boundary corresponding to the obstacle distribution parameters, and determine the minimum value of each distance as the relative distance.

3. The UAV dynamic obstacle avoidance method based on improved reinforcement learning according to claim 1, characterized in that, In step S2, constructing the initial state vector specifically includes the following steps: Obtain preset distance normalized reference values, tension normalized reference values, and flight parameter normalized reference values; The relative distance, the tension change rate, the current tension, and the current flight parameters are normalized using the corresponding normalized reference values ​​to generate corresponding normalized distance features, normalized tension change features, normalized current tension features, and normalized flight features. The normalized distance feature, the normalized tension change feature, the normalized current tension feature, and the normalized flight feature are combined in a preset sequence to generate an initial state vector.

4. The UAV dynamic obstacle avoidance method based on improved reinforcement learning according to claim 1, characterized in that, In step S3, the output temporal enhancement feature vector specifically includes the following steps: Arrange the historical state vector and the initial state vector in chronological order to construct a temporal state feature matrix; The time-series state feature matrix is ​​multiplied by a preset query weight matrix, key weight matrix, and value weight matrix, respectively, to generate the corresponding query matrix, key matrix, and value matrix. Calculate the inner product of the query matrix and the key matrix, and process the result of the inner product using a normalized exponential function to generate an attention weight matrix; Calculate the product of the attention weight matrix and the value matrix, and output the temporal enhancement feature vector.

5. The UAV dynamic obstacle avoidance method based on improved reinforcement learning according to claim 4, characterized in that, The process of calculating the inner product of the query matrix and the key matrix, and processing the result of the inner product using a normalized exponential function to generate the attention weight matrix, specifically includes the following steps: The relative distance and the rate of change of tension are extracted from the state vectors at each time step of the time-series state feature matrix, and normalized respectively to eliminate physical dimensions. The difference between the value 1 and the normalized relative distance is taken as the distance risk sub-item, and the absolute value of the normalized tension change rate is taken as the tension risk sub-item. The distance risk item and the tension risk item are linearly weighted using preset weighting coefficients to obtain the risk coefficient at each time step, and a time-series bias matrix is ​​constructed based on the difference between the risk coefficients at adjacent time steps. The attention weight matrix is ​​generated by adding the inner product of the time-series bias matrix, the query matrix, and the key matrix, and then performing a normalized exponential operation on the added result.

6. The UAV dynamic obstacle avoidance method based on improved reinforcement learning according to claim 1, characterized in that, In step S4, the output, which includes the initial motion vector of flight speed adjustment, heading adjustment, and retraction / deployment control, specifically comprises the following steps: The temporal enhancement feature vector is input into the fully connected hidden layer of the reinforcement learning policy network and processed by the activation function to output the hidden layer action features; The hidden layer action features are input into the action output layer of the reinforcement learning policy network, which outputs a continuous normalized action vector. Obtain the preset flight control physical constraint range and the retraction control physical constraint range, and use the flight control physical constraint range and the retraction control physical constraint range to perform denormalization mapping on the normalized action vector, separate the flight speed adjustment amount, the heading adjustment amount and the retraction device control amount, and combine them to generate the initial action vector.

7. The UAV dynamic obstacle avoidance method based on improved reinforcement learning according to claim 1, characterized in that, In step S5, generating the final collaborative instruction specifically includes the following steps: When it is determined that the current tension is greater than a preset tension threshold or the relative distance is less than a preset distance threshold and there is a risk of exceeding the limit, a preset speed attenuation coefficient is obtained, and the flight speed adjustment amount is multiplied by the speed attenuation coefficient to generate a corrected speed amount; The offset compensation angle in the direction away from the obstacle is calculated based on the obstacle distribution parameters in the initial state vector, and the offset compensation angle is superimposed on the heading adjustment amount to generate the corrected heading vector; The corrected velocity, the corrected heading vector, and the control quantity of the launch and recovery device are concatenated to generate the final coordinated command.

8. The UAV dynamic obstacle avoidance method based on improved reinforcement learning according to claim 7, characterized in that, The calculation of the offset compensation angle in the direction away from the obstacle based on the obstacle distribution parameters in the initial state vector specifically includes the following steps: Two candidate offset compensation angles are generated on both sides of the obstacle based on the obstacle distribution parameters, and then combined with the flight speed adjustment, the heading adjustment and the retraction device control to obtain two candidate motion vectors. The candidate position of the UAV and the candidate release length of the flexible connector for the next control cycle are calculated based on each candidate motion vector, and the candidate tension of the flexible connector and its candidate relative distance to the obstacle are calculated using the catenary model. Candidate risk values ​​are calculated based on the deviation of the candidate tension from the preset tension threshold and the deviation of the candidate relative distance from the preset distance threshold, respectively. The candidate offset compensation angle corresponding to the candidate risk value with the smaller value is determined as the offset compensation angle.

9. The UAV dynamic obstacle avoidance method based on improved reinforcement learning according to claim 1, characterized in that, S6 specifically includes the following steps: The final collaborative command is parsed to separate the UAV pose control data packet and the retractor drive data packet; Using a preset communication protocol, the UAV pose control data packet is converted into a flight control electrical signal, and the retraction device drive data packet is converted into a motor control electrical signal; The flight control electrical signals are transmitted to the flight controller via a communication bus, and the motor control electrical signals are transmitted to the main control board of the take-up and take-down device.

Images