Unmanned aerial vehicle exploration navigation method based on sequence enhancement improved SAC algorithm

Abstract

This invention belongs to the field of UAV navigation technology and discloses a UAV exploration and navigation method based on a sequence enhancement improved SAC algorithm. The method includes the following steps: acquiring multimodal perception data; forming standardized state vectors from the multimodal perception data at each time step; inputting the state vectors from each time step into a state sequence into an improved Transformer network for feature extraction to obtain high-dimensional feature vectors; inputting the high-dimensional feature vectors into a teacher-student improved SAC network, where the student network outputs the original action; determining whether the distance to the nearest obstacle is less than a threshold; if so, correcting the original action with a VO safety shield to obtain a safe action, projecting / clipping it to a dynamically feasible set, and then executing it. This invention enables UAVs to achieve safe and efficient autonomous exploration and flight in unknown environments.

Description

Technical Field

[0001] This invention belongs to the field of UAV navigation technology, specifically relating to a UAV exploration and navigation method based on a sequence enhancement-improved SAC algorithm. Background Technology

[0002] In recent years, with the rapid development and widespread application of unmanned aerial vehicle (UAV) technology, UAVs have played an increasingly important role in diverse tasks such as urban delivery, inspection, and search and rescue. However, during mission execution, UAVs typically need to autonomously navigate and avoid obstacles in low-altitude, large-scale, and dynamically complex environments. Improving their environmental adaptability and autonomous decision-making capabilities has become a research hotspot and challenge. Traditional navigation and path planning methods, such as A... While search algorithms, Dijkstra's algorithm, artificial potential field method, genetic algorithm, ant colony optimization algorithm, etc. can achieve basic path planning in static or prior known environments, when faced with complex, variable and unknown three-dimensional space, these methods often suffer from problems such as reliance on prior information, easy getting trapped in local optima, high computational complexity and weak generalization ability, making it difficult to meet the needs of multi-degree-of-freedom UAVs for real-time obstacle avoidance and dynamic autonomous exploration.

[0003] In recent years, deep reinforcement learning (DRL) technology has provided new ideas for autonomous navigation of unmanned aerial vehicles (UAVs) in unknown environments due to its ability to learn optimal policies autonomously through interaction with the environment. Among them, Soft Actor-Critic (SAC), a deep reinforcement learning algorithm based on maximum entropy theory, has shown good performance in enhancing exploration capabilities and improving policy stability, and has been gradually applied to UAV autonomous navigation and obstacle avoidance tasks. However, some limitations still exist in practical applications: on the one hand, the state-action space grows exponentially in complex dynamic environments, and traditional SAC usually uses a simple multilayer perceptron (MLP) as a feature extractor, which is difficult to effectively capture long-range dependencies in state sequences; on the other hand, the SAC network relies on a large number of state-action pairs for trial-and-error learning, the SAC algorithm requires a large number of environmental interaction samples, the convergence speed is slow, and the policy is prone to getting trapped in local optima when facing complex environments, resulting in limited generalization ability.

[0004] Therefore, how to further improve the perception capabilities and decision-making efficiency of UAVs in complex and ever-changing environments during autonomous exploration and navigation has become a key issue that urgently needs to be addressed. Summary of the Invention

[0005] To address the technical problems of insufficient perception capabilities and decision-making efficiency in existing UAV autonomous navigation methods, this invention proposes a UAV exploration and navigation method based on a sequence enhancement-based improved SAC algorithm. By introducing an efficient feature extraction network and diversified decision-making mechanisms, the method enhances the UAV's adaptability to the environment and the robustness of path planning. At the same time, the SAC algorithm is improved using a teacher-student framework to enable the UAV to conduct safe and efficient autonomous exploration and flight in unknown environments.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a UAV exploration and navigation method based on sequence enhancement improved SAC algorithm, comprising the following steps:

[0007] Step 1: Acquire multimodal sensing data, and form a standardized state vector from the multimodal sensing data at each time step, i.e.:

[0008] ;

[0009] in, Indicates time t The state vector, This represents the internal state vector of the UAV; Represents a static obstacle vector. Represents a dynamic obstacle vector. This represents the boundary vector of the unknown region detected by the local map;

[0010] Step 2: Form a state sequence from the state vectors of each time step. Inputting the improved Transformer network for feature extraction yields high-dimensional feature vectors. The improved Transformer network includes a position encoder, a multi-layer Transformer encoder, a pooling layer, and a concatenation layer. The position encoder encodes the state vector to obtain a sequence matrix. The multi-layer Transformer encoder calculates the time-series hidden representation matrix based on the position-encoded sequence matrix. The pooling layer aggregates the time-series hidden representation matrix. The concatenation layer concatenates the feature representation output from the pooling layer with the state vector output from the position encoder to obtain a high-dimensional feature vector. ;

[0011] Step 3: Convert the high-dimensional feature vector The input is a teacher-student improved SAC network, which includes a teacher network and a student network. During the training phase, the teacher network is trained using a standard SAC network and provides knowledge distillation signals. The student network inherits the teacher's policy through distillation and combines distillation loss, low-rank reconstruction regularization, attention direction consistency loss, and VO safety triggering loss to form the total loss. During the deployment phase, only the student network is retained for online inference, and the student network outputs the original actions. ;

[0012] Step 4: Determine if the distance to the nearest obstacle is less than a threshold. If so, then execute the action output by the student network. The corrected action is obtained by performing VO security shield correction, and then projected / trimmed into the dynamic feasible set before being sent out for execution; otherwise, the original action output by the student network is executed directly.

[0013] In step 1, the method for obtaining the boundary vector of the unknown region detected by the local map specifically includes the following steps:

[0014] Step 1.1: Extract all preface points based on the local 3D occupancy raster map;

[0015] Step 1.2: For each detected preamble point, calculate the information gain; the calculation formula is:

[0016] ;

[0017] in, Indicates the frontier point The corresponding information gain; Indicates the frontier point The corresponding local information gain; and These are the weighting coefficients; Indicates the current position. This indicates the cost or penalty associated with distance;

[0018] Step 1.3: Use the leading edge point and its corresponding information gain as the boundary vector of the unknown region detected by the local map. .

[0019] In step 2, the displacement encoder processes the state sequence. The method for position encoding is as follows:

[0020] Calculate the confidence-weighted state embedding vector :

[0021] ;

[0022] in, This represents the confidence-weighted state embedding vector. , The parameters represent the linear mapping; This represents the activation / regularization function. This represents a channel-by-channel differentiable scaling function. Indicates element-wise multiplication;

[0023] Calculate the position-encoded state vector:

[0024] ;

[0025] in, This represents a learnable time-decayed positional encoding. Let L represent the confidence-weighted state embedding vector at time ti, and L represent the length of the state sequence.

[0026] In step 2, the improved Transformer network used includes a multi-layer Transformer encoder comprising two stacked Transformer coding layers. Each Transformer coding layer includes an 8-head self-attention mechanism, a feedforward network, a normalization layer, and an output layer connected in sequence.

[0027] In the aforementioned teacher-student improved SAC network, the loss function of the student network is:

[0028] ;

[0029] in, , , and These represent knowledge distillation losses, respectively. Refactoring regular expressions Prior loss of attentional direction consistency Loss triggered by VO Security Shield The corresponding weights.

[0030] Refactoring regular expressions The calculation formula is:

[0031] ;

[0032] in, The projection matrix of the Transformer encoder is represented by U and V, which are the two matrices decomposed from the projection matrix W. This represents the Frobenius norm.

[0033] VO Security Shield Trigger Loss The calculation formula is:

[0034] ;

[0035] in, Indicates the average of experience; This represents the original action by which the teacher and student improve the output of the SAC network. With candidate actions The regularized square, Indicates an indicator function.

[0036] In step 4, the output action of the improved SAC network is... The method for modifying VO Security Shield is as follows:

[0037] Step 4.1: Determine the speed obstacle set for each obstacle. The calculation formula is as follows:

[0038] ;

[0039] in, Represents the set of speed obstacles for the j-th dynamic obstacle; and Let represent the estimated position and velocity of the j-th dynamic obstacle at the current moment, respectively. Indicates time The location of the drone v Indicates candidate velocity, This indicates a time less than the prediction time window, where T represents the prediction time window. Indicates the safety threshold;

[0040] Step 4.2: Determine the safe speed set based on the speed set of all obstacles. ,in, ; Represents a three-dimensional set. Represents the union;

[0041] Step 4.3: Obtain the original actions for the teacher-student improved SAC network output. Determine if it belongs to the safe speed set; if so, correct the action. Values If it does not belong to the set of safe speeds, then it is included in the set of safe speeds. Searching for and Minimum distance and belongs to the dynamically feasible set The candidate velocity v is used as the correction action ,Right now:

[0042] ;

[0043] in Let v denote the dynamically feasible set, and v denote the safe velocity set. Candidate velocity in the process.

[0044] In step 4, the dynamic feasible set The calculation formula is:

[0045] ;

[0046] in This is the maximum flight speed of the drone; This is the maximum acceleration; The actual flight speed in the previous time step. Indicates the control cycle.

[0047] The UAV exploration and navigation method based on sequence enhancement and improved SAC algorithm further includes the following steps:

[0048] A three-dimensional simulation environment is constructed, in which a UAV is equipped with a high-sensitivity depth camera, a three-dimensional LiDAR, and an inertial measurement unit;

[0049] Simulations are performed in a 3D simulation environment to acquire multimodal sensing data;

[0050] Steps 1-4 are performed based on multimodal sensing data to train the improved Transformer network and the teacher-student improved SAC network.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] (1) This invention uses the Transformer network as the core of feature extraction. Its self-attention mechanism can effectively capture long-range dependencies in the state sequence and generate feature representations with rich temporal information. Compared with traditional recurrent neural networks or convolutional networks, the Transformer shows stronger modeling ability when processing long sequence data, providing a more comprehensive and accurate environmental state representation for the subsequent decision network, and significantly improving the system's ability to perceive and understand complex dynamic environments.

[0053] (2) By introducing a teacher-student framework, this invention separates the exploration and optimization processes. The teacher network conducts thorough environmental exploration and accumulates high-quality experience, while the student network quickly learns the essence of the teacher network's strategies through a knowledge distillation mechanism. This design reduces the environmental interaction requirements by about 50-70%, significantly reduces training costs and time, and ensures the quality of strategies. It is particularly suitable for practical application scenarios with high sample efficiency requirements.

[0054] (3) This invention employs a triple knowledge distillation mechanism of strategy, value function, and Q function to ensure that the student network can fully inherit the excellent characteristics of the teacher network. This multi-level distillation not only transmits the final policy behavior but also the value judgment standard, enabling the student network to maintain its lightweight nature while still achieving a task success rate and obstacle avoidance performance comparable to the teacher network in simulation tests. This effectively suppresses policy degradation and improves decision-making stability and flight safety margin in complex dynamic environments.

[0055] In summary, this invention can significantly improve the navigation safety and efficiency of unmanned aerial vehicle (UAV) systems. Attached Figure Description

[0056] Figure 1 A flowchart illustrating an UAV exploration and navigation method based on a sequence enhancement-improved SAC algorithm, provided in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of drone projection in an embodiment of the present invention, wherein (a) is a top view of the horizontally projected light ray with a projection angle of 360°, and (b) is a side view of the vertical plane ray.

[0058] Figure 3 This is a schematic diagram of the multimodal environment perception and Frontier point selection process in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of the improved Transformer feature extractor and the improved teacher-student SAC network in an embodiment of the present invention;

[0060] Figure 5 This invention relates to the Velocity Obstacle security shield correction and closed-loop training execution process. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] This invention provides a UAV exploration and navigation method based on a sequence enhancement-based improved SAC algorithm, deeply embedding a multi-layer Transformer encoder structure into the SAC network. This invention constructs an input tensor containing historical state sequences and utilizes the multi-head self-attention mechanism in the multi-layer Transformer encoder to accurately characterize spatiotemporal dependencies and extract high-dimensional feature vectors. These vectors are then input into the SAC network to achieve UAV action output, effectively overcoming the bottlenecks of traditional SAC networks, such as insufficient utilization of temporal information, slow policy convergence, and susceptibility to local optima. This improvement aims to accelerate policy convergence during training and enhance global optimum search capability. Experimental results show that after inputting high-dimensional feature vectors containing spatiotemporal dependencies into the SAC network, it can more quickly perceive movement trends and respond in dynamic obstacle environments, while significantly enhancing the robustness of the navigation strategy and task completion efficiency.

[0063] Furthermore, this invention proposes a series of supporting optimization measures to further improve the system's perception accuracy and flight safety. First, a multimodal environmental perception subsystem is constructed, integrating a RealSense D435i depth camera with 3D LiDAR point clouds, and combining target detection, top-down depth mapping, and spatial clustering algorithms to achieve high-precision 3D occupancy mesh construction for both static and dynamic obstacles. Second, an exploration-driven mechanism based on Frontier information gain and trajectory smoothing regularization is introduced into the reward function to achieve a dynamic trade-off between exploration efficiency and flight safety. Collision penalties are constructed separately for static and dynamic obstacles, and the path smoothing term uses speed change as a metric to reduce the risks and energy consumption caused by rapid acceleration and deceleration. In addition, a teacher-student distillation-style improved SAC framework is proposed at the SAC policy level: Phase A uses standard SAC for pre-training and freezes the teacher model; Phase B involves continuous interaction between the student network and the teacher network, guided by the teacher's output, jointly minimizing the total loss and completing online updates. The aforementioned supporting measures and improvements collectively deliver quantifiable benefits: while maintaining strategy quality, they significantly reduce environmental interaction, accelerate convergence, and enhance generalization and safety, achieving a complete closed-loop system capable of both proactively exploring unknown spaces and dynamically avoiding obstacles. On the action output side, the strategy employs a Gaussian distribution with Tanh limiting, combined with a minimum deviation mechanism using dual Critic and V-networks, improving valuation stability and consistency of output with flight control constraints. Combined with the VO minimum intervention action correction method during deployment, this provides a "last line of defense" for the strategy, ensuring smooth and safe flight in unknown dynamic environments.

[0064] The aforementioned supporting measures and improvements collectively bring quantifiable benefits: while maintaining the quality of the strategy, the amount of environmental interaction is significantly reduced, convergence is accelerated, and generalization and safety are improved. Furthermore, as students gradually learn to generate "feasible" actions, the frequency of online VO intervention decreases significantly as training progresses, thereby achieving a complete closed-loop system that can both actively explore unknown spaces and dynamically avoid obstacles.

[0065] Specifically, such as Figure 1 As shown, the UAV exploration and navigation method based on the sequence enhancement improved SAC algorithm provided in this embodiment of the invention includes the following steps:

[0066] Step 1: Acquire multimodal sensing data, and form a standardized state vector from the multimodal sensing data at each time step, i.e.:

[0067] ; (1)

[0068] in, Indicates time t The state vector, This represents the internal state vector of the drone, including the drone's current position. Current heading of the drone UAV velocity vector Target location of the drone ; Represents a static obstacle vector. Represents a dynamic obstacle vector. This represents the boundary vector of the unknown region detected by the local map.

[0069] Specifically, in this embodiment, multimodal sensing data is acquired through an RGB-D camera, a three-dimensional LiDAR, and an inertial measurement unit.

[0070] Specifically, step 1 of this embodiment further includes obtaining the boundary vector of the unknown region detected by the local map. Methods, such as Figure 2-3 As shown, the specific steps include:

[0071] Step 1.1: Extract all frontier points based on the local 3D occupancy grid map; in this embodiment, the classic Frontier extraction method is used:

[0072] ; (2)

[0073] in, For locally occupied grid sets; Free Given a set of free space lattice points; Unknown For an unknown set of spatial grid points; Indicates the front detection threshold; Indicates the frontier point A set; Indicates and Adjacent sets of voxels / lattice points, Indicates the frontier point A grid point within a neighborhood (Note: here) This refers to the voxel / mesh index adjacent to the front candidate point, used to determine whether an unknown neighborhood exists. Indicates the frontier point The corresponding number of observations. (Based on the number of observations) Small-scale confidence levels or observation counts (optional) that can remove noise can be used to control detection robustness.

[0074] Step 1.2: Calculate the information gain for each detected leading edge point; the calculation formula is:

[0075] ; (3)

[0076] in, Indicates the frontier point The corresponding information gain; Indicates the frontier point The corresponding local information gain; and These are the weighting coefficients; Indicates the current position; Indicates the frontier point Current location of the drone Distance-related cost penalties (such as Euclidean distance or path cost) are used to suppress the unreasonable influence of distant candidate points on immediate decisions, thereby achieving a balance between exploration information gain and action cost.

[0077] The formula for calculating local information gain is:

[0078] ; (4)

[0079] in, For the forefront The set of voxels in the local neighborhood centered on the voxel window; voxels within the neighborhood The known probability (of something being observed / occupied or free); Voxel area or voxel volume weights, such as weighting voxels under projection or different sensor resolutions.

[0080] Step 1.3: Use the calculated frontier points and corresponding information gain values ​​as the boundary vectors of the unknown regions detected by the local map. .Right now:

[0081] ; (5)

[0082] in Indicates the first A cutting-edge point, Indicates the first Information gain corresponding to each frontier point This represents the number of leading edge points. To facilitate subsequent neural network input, this set can be fixed in length.

[0083] In this embodiment, a cost penalty is added to the local information gain corresponding to candidate points far from the current drone position. Information gain obtained Then information gain With the forefront The resulting unknown region boundary vector as a state vector This system, as a component, balances readily available information gathering with exploration incentives, thereby avoiding the frequent and costly selection of remote frontiers. This allows drones to prioritize nearby frontiers that offer substantial new information during decision-making.

[0084] The final state vector The input is improved by extracting sequence features from the Transformer network, which enables the subsequent teacher-student improved SAC network to introduce exploratory driving (information gain driving) and balance safety and cost when generating actions.

[0085] Step 2: Form a state sequence from the state vectors generated at each time step. Inputting the improved Transformer network for feature extraction yields high-dimensional feature vectors. .in, Representing time respectively t- L , t-L+ 1… t The corresponding state vector. The improved Transformer network includes a position encoder, a multi-layer Transformer encoder, a pooling layer, and a stitching layer.

[0086] The position encoder is used to encode the state vector to obtain a sequence matrix. The multi-layer Transformer encoder is used to calculate the time-by-time hidden representation matrix based on the position-encoded sequence matrix. The pooling layer is used to pool and aggregate the time-by-time hidden representation matrix. The concatenation layer is used to concatenate the feature representation output by the pooling layer with the state vector output by the position encoder to obtain a high-dimensional feature vector. .

[0087] To enable the Transformer to accept serialized state vectors while also considering temporal order information, this embodiment performs unified dimensional mapping and position encoding on the original state vectors before feeding them into the multi-layer Transformer encoder. To improve the responsiveness of sequence features to short-term dynamics and sensor confidence changes, the position encoder in this invention employs learnable time-decayed position encoding and weighted embedding mapping.

[0088] First, linear embedding is performed on each state vector and the sensor confidence level is introduced. (Estimated by the sensor module) Related scaling transformation:

[0089] ; (6)

[0090] in, This represents the confidence-weighted state embedding vector. , The parameters represent the linear mapping; This represents the activation / regularization function (e.g., LayerNorm + GELU). This represents a channel-wise differentiable scaling function (which can be implemented as a small network or a channel-wise affine transformation) to reduce the impact of confidence observations on subsequent attention. This indicates element-wise multiplication.

[0091] Then, a learnable time-decay position encoding is defined. :

[0092] ; (7)

[0093] in, The reference position vector can be represented by sin / cos or initialized as a Gaussian random vector. Indicates time step i Learnable gating vectors that decay (i.e., relative time intervals). Represents an element-wise Sigmoid. This is achieved through gating. The following Attenuation automatically suppresses the influence of older historical steps on the current attention. The position-encoded state vector is obtained by adding the attenuated position code to the confidence-weighted embedding vector, i.e.:

[0094] ; (8)

[0095] Indicates the first ti The state vector encoded by the position at each time step. Indicates the first tiThe confidence-weighted state embedding vector at each time step. The sequence matrix after position encoding by the position encoder. It can be represented as:

[0096] ; (9)

[0097] in, This represents the dimension of each vector in the sequence matrix. L Indicates the number of time steps; the position-encoded sequence matrix. The input is then fed into a multi-layer Transformer encoder for feature extraction.

[0098] This invention significantly improves robustness and decision stability in dynamic scenarios (short-term sudden obstacles, sensor frame drops) by introducing sensor confidence scales and time decay gating. This provides a more reliable input distribution for subsequent attention modules.

[0099] The multi-layer Transformer encoder in this embodiment consists of multiple stacked Transformer coding layers, each including a multi-head self-attention mechanism and a feedforward fully connected network. Specifically, it includes two stacked Transformer coding layers, each including an 8-head self-attention mechanism, a feedforward network, a normalization layer, and an output layer connected in sequence.

[0100] In existing multi-layer Transformer encoders, dynamic obstacles... Continuous data can be obtained by tracking / associating within the historical window. The average displacement vector of the differences between adjacent positions ,Right now:

[0101] (10)

[0102] in and For dynamic obstacles At any moment and time Spatial location. The average displacement vector. Normalization yields the normalized average displacement vector as follows: .

[0103] In this embodiment, the multi-layer Transformer encoder retrieves the attention row of the current query to the historical key from the specified attention layer. Attention coefficients are obtained by normalization within the index set corresponding to dynamic obstacles. Constructing attention-weighted perceptual displacement ,Right now:

[0104] ; (11)

[0105] in, and These represent the j-th dynamic obstacle at time t and time t, respectively. Spatial location, weighted displacement for attention Normalization is performed to obtain the normalized weighted sensing displacement. .

[0106] Furthermore, in the multi-layer Transformer encoder of this embodiment, the normalized weighted sensing displacement is weighted according to the obstacle importance weight. By performing weighted and normalized calculations, the overall directional loss is obtained. :

[0107] ;(12)

[0108] in, This represents the importance weight of the j-th dynamic obstacle.

[0109] Furthermore, in the Transformer encoder, both the self-attention layer and the feedforward network contain several linear projection matrices, whose weights are collectively referred to as projection matrices. W The multi-layer Transformer encoder of this application uses low-rank methods for the above projection matrices during calculation. r Decomposition, approximating each projection matrix as the product of two matrices:

[0110] ; (13)

[0111] Here, U and V represent the two matrices after decomposition. , , , r < d; Low-rank decomposition approximation can reduce the parameter and multiplication complexity of large matrices in the Transformer encoder, reducing the computational cost from the original Reduced to decomposed O represents the order of magnitude (order).

[0112] Furthermore, in the multi-layer Transformer encoder, a reconstruction error term is introduced as a reconstruction regularization to control the decomposition error, and its expression is:

[0113] ;(14)

[0114] in, This indicates a reconstruction regularization. Subsequent network training will incorporate this regularization. Incorporate into total loss and use coefficient Weighted, This represents the Frobenius norm.

[0115] The sequence input is fed into a multi-layer Transformer encoder, and the output of the multi-layer Transformer encoder is:

[0116] ; (15)

[0117] The time-by-time hidden representation matrix representing the output of a multi-layer Transformer encoder. The time-by-time hidden representation matrix will be used. By using a pooling layer for average pooling, the information from an entire historical period can be compressed into a single vector, reducing noise and redundancy and highlighting stable trends. The features aggregated by average pooling in the pooling layer can be represented as follows:

[0118] ; (16)

[0119] The concatenation layer represents the feature representation output by the pooling layer. The state vector encoded with the current position The concatenation yields a high-dimensional feature vector. ,Right now:

[0120] ; (17)

[0121] in, This represents the high-dimensional feature vector output by the improved multi-layer Transformer encoder, which subsequently serves as the state input to the teacher-student improved SAC network. This is achieved by combining the aggregated feature representation using average pooling with the current state vector. Connecting information is equivalent to combining "historical information" with "current details," preserving the temporal context without over-reliance on distant history.

[0122] In step 2, the multi-layer Transformer network used includes two stacked Transformer coding layers. Each Transformer coding layer includes an 8-head self-attention mechanism, a feedforward network, a normalization layer, and an output layer connected in sequence.

[0123] The beneficial effects of this embodiment are mainly reflected in the overall improvement of the representation, aggregation, and modeling of sequence information: First, step 2 introduces confidence scaling and learnable time decay gating before conventional position encoding, so that distant historical and low-confidence observations are automatically downweighted, making the input distribution more robust; then, the position-encoded historical sequence is fed into the improved multi-layer Transformer encoder, and two structured constraints are added to the Transformer encoder—attention direction consistency loss and low-rank decomposition of the projection matrix + reconstruction regularization; then, average pooling is performed on the time-by-time hidden vector to form a temporal summary, which is concatenated with the current step embedding and linearly aligned to obtain a high-dimensional feature vector. As the final state fed into the SAC network, compared to the conventional approach of "only taking the output at the last moment", this design of "confidence / time gating → encoder direction prior and low-rank acceleration → temporal pooling + current detail splicing" can suppress distant historical and low-confidence noise, highlight recent high-confidence cues, and improve forward perception of dynamic obstacles and policy stability. At the same time, low-rank decomposition effectively reduces the number of parameters and inference latency, which is more conducive to real-time deployment in unknown dynamic environments.

[0124] Step 3: Input the high-dimensional feature vector into the teacher-student improved SAC network; during the training phase, the teacher network is trained using the standard SAC and provides knowledge distillation signals, while the student network inherits the teacher network's strategy through distillation, and the total loss is formed by combining distillation loss, low-rank reconstruction regularization, attention direction consistency loss, and VO safe triggering loss; during the deployment phase, only the student network is retained for online inference, and the student network outputs the original actions. .

[0125] In this embodiment, the student network combines distillation loss, low-rank reconstruction regularization, attention direction consistency loss, and VO safety triggering loss to form the total loss without changing the basic update formula of the SAC algorithm.

[0126] During training, the teacher network is updated following the conventional methods of existing SAC algorithms, such as... Figure 4 As shown. For autonomous exploration tasks, a multi-component reward function is defined, comprehensively considering: safety and obstacle avoidance, local information gain, and task objectives. The immediate reward function used by the teacher-student improved SAC network when interacting with the environment is:

[0127] ; (18)

[0128] in, Indicates an immediate reward. and These represent the collision penalties for static and dynamic objects, respectively. Indicates the current location of the drone The corresponding local information gain, The weighting coefficient is non-negative. As a path smoothing reward item, in this embodiment, the reward will be given immediately. The reward signal of the SAC algorithm is used to construct the target values ​​of the Q-network and V-network, thus forming the basic total loss including Q-loss, V-loss, policy loss, and temperature parameter loss. As the loss function of the teacher network, the formulas for calculating collision penalties for static and dynamic obstacles are:

[0129] ; (19)

[0130] ; (20)

[0131] in, , These represent the number of rays on the vertical and horizontal planes, respectively. These represent the vertical and horizontal indices of the ray, respectively. Indicates the first Observation / occupancy probability or status indication of static obstacles along the ray direction. It is the numerical stability constant. This indicates the number of dynamic obstacles currently detected. Indicates the center position of a dynamic obstacle.

[0132] The formula for calculating the path smoothing reward is:

[0133] ; (twenty one)

[0134] in, and Indicates the current time step and the previous time step The corresponding drone velocity vector.

[0135] In this embodiment, the value update and policy update of the SAC algorithm adopt the standard form, including the updates of V-Critic, dual Q-Critic and temperature parameters.

[0136] In this embodiment, the standard RL objective of the SAC algorithm is jointly optimized with constraints such as knowledge distillation, low-rank reconstruction, directional consistency, and VO security shield penalty in the student network.

[0137] Specifically, in this embodiment, without changing the basic update formula of the SAC algorithm, the student network's basic total loss under the standard RL objective is... The following constraints are introduced: ① Knowledge distillation loss from the teacher network ② Reconstruction regularization introduced by the improved low-rank decomposition of attention projection in multi-layer Transformer encoders ③ Prior loss of attentional direction consistency ④VO security shield trigger loss Its weighted sum is denoted as the constraint loss of the student network. :

[0138] ; (twenty two)

[0139] in, , , and These represent knowledge distillation losses, respectively. Refactoring regular expressions Prior loss of attentional direction consistency Loss triggered by VO Security Shield The corresponding weights. The total loss function of the teacher-student improved SAC network is:

[0140] ; (twenty three)

[0141] Specifically, the training process of the teacher-student improved SAC network includes two stages: a teacher training stage and a student distillation and joint optimization stage, with a simulation interaction / replay pool as a prerequisite. Specifically, in the teacher training stage, the teacher network is first independently trained to a stable state using the standard SAC, and then frozen to provide distillation supervision signals. In the student distillation and joint training stage, the student network continues to interact with the environment and samples gradients from the replay pool for updates; at each update step, the total loss function of the student network is calculated using formula (23).

[0142] In step 3, by inputting the global features extracted by the improved multi-layer Transformer encoder into the teacher-student improved SAC network and utilizing the teacher-student distillation training mechanism, the model complexity is significantly reduced while maintaining high-performance policies, thereby improving real-time inference efficiency. Compared with existing SAC algorithms that rely solely on convolutional or recursive structures, this invention can converge faster and output actions stably in dynamic environments, exhibiting stronger environmental adaptability and real-time deployment capabilities.

[0143] Step 4: Determine if the distance to the nearest obstacle is less than a threshold. If so, adjust the original action of the teacher-student improved SAC network output. Perform Velocity Obstacle (VO) shield correction to obtain the correction action after shield correction. After projecting / clipping it to the dynamically feasible set, it is sent out for execution. Otherwise, the original action of the teacher-student improved SAC network output is executed directly, such as... Figure 5 As shown.

[0144] Specifically, in step 4, the output actions of the teacher-student improved SAC network are... The method for modifying VO Security Shield is as follows:

[0145] Step 4.1: Determine the speed obstacle set for each obstacle, where let the first obstacle be... The estimated positions and velocities of the dynamic obstacles at the current moment are as follows: and It can be given by forming a standardized state vector *st* from multimodal sensing data at each time step; the prediction time window is... If the drone is at the candidate speed During flight, at a certain time The distance to one of the dynamic obstacles will be less than the safety threshold. If the candidate velocity falls into the velocity obstacle set of the dynamic obstacle, then the velocity obstacle set can be represented as:

[0146] ; (twenty four)

[0147] in, Indicates the first j A set of speed obstacles for dynamic obstacles. For a moment The location of the drone can be determined by its state vector. Obtain. This indicates the time shorter than the prediction time window. For the first j Linear prediction of obstacle locations.

[0148] Step 4.2: Combine the speed obstacles of all obstacles and their complements to obtain the safe speed set. ,Right now:

[0149] ; (25)

[0150] The above formula represents the three-dimensional velocity space. In the middle, after removing the union of the speed obstacle sets of all obstacles, what remains is the set of allowed safe speeds. , It represents the union of sets.

[0151] Step 4.3: Under dynamic and control constraints, find the feasible action that is closest to the original action from the safety domain.

[0152] The specific method involves obtaining the original actions of the teacher-student improvement SAC network output. Determine whether it belongs to the safe speed set. Then correct the action. Values .like Then search for a match in the safe speed set. The candidate velocity v with the smallest distance and satisfying the dynamic constraints is used as the correction action. ,Right now:

[0153] ; (26)

[0154] in, Describing the dynamically feasible set, we have:

[0155] ;(27)

[0156] in This is the maximum flight speed of the drone; The maximum permissible acceleration; This is the actual flight speed at the previous time step (the speed after the safety shield correction). This is expressed as control period / sampling time. The control period represents the time step between two adjacent velocity updates; the prediction time window... T This usually represents the total time for collision prediction, typically taken as the control period. Integer multiples of, for example , where M is a positive integer.

[0157] When performing a security shield correction, an experience tuple should be recorded in the experience replay pool. The experience tuple is: During the training phase, the safety shield correction is used as a soft constraint for reinforcement learning training: VO safety shield trigger loss is introduced during the student phase. and with coefficient Adding this to the total loss, the VO security shield trigger loss is:

[0158] ; (28)

[0159] in, Indicates the average of experience; This represents the original action by which the teacher and student improve the output of the SAC network. With candidate actions The regularized square. It is an indicator function if and only if the original action is not in the VO safe set, i.e. When a VO security fix is ​​triggered, the indicator function takes the value of Otherwise, its value is Therefore, this penalty only applies when a safety correction occurs. Only then will it be counted as a loss triggered by VO Security Shield. This allows the learning strategy to gradually generate more "feasible" actions while ensuring safety, reducing the frequency of online corrections.

[0160] The UAV exploration and navigation method based on the sequence augmentation improved SAC algorithm in this embodiment also includes a step of training the network, and the training process specifically includes the following sub-steps:

[0161] (1) Constructing a high-fidelity 3D simulation and sensor model

[0162] In this embodiment, a 3D simulation training environment is constructed based on the NVIDIA Isaac Sim and ROS platform. The simulation scene is a hybrid indoor / outdoor structure, including various static obstacles (walls, furniture, columns, etc.) and dynamic obstacles with configurable behaviors (pedestrians, mobile robots, vehicles, etc.). During simulation initialization, the starting pose, target point, and task boundaries are randomly set to achieve task randomization. The airborne multi-sensor model includes an RGB-D camera, a 3D LiDAR, and an IMU. Multimodal data streams are generated according to the actual sensor frame rate and transmitted in real time through the ROS interface for use by downstream perception and learning modules.

[0163] (2) Multimodal perception and lightweight target tracking preprocessing

[0164] Simultaneously process camera and LiDAR data at each time step:

[0165] RGB-D camera: Identifies targets in RGB images using a lightweight detector (YOLO) and generates 3D bounding boxes by U-depth projection of the depth map, providing category and coarse spatial information.

[0166] 3D LiDAR: Outputs high-resolution point clouds and uses DBSCAN clustering to accurately identify the location and size of obstacle clusters, compensating for the shortcomings of cameras at long distances and under occlusion conditions.

[0167] Fusion and Tracking: Consistent matching and fusion of the two types of results are performed to maintain a static occupied voxel map updated by recursive Bayes; continuous tracking and velocity estimation are achieved for dynamic targets based on Kalman filtering and feature similarity to obtain state variables such as target position, velocity and importance weight.

[0168] State construction and serialization: at time t The local occupancy probability, dynamic target state (position / velocity / importance weights), and confidence scores of each sensor are encoded and aggregated into an original state vector. ; Form a state sequence according to the historical window length .

[0169] (3) Sequence enhancement feature extraction and strategy training

[0170] After position encoding, the state sequence is input into an improved multi-layer Transformer encoder for feature extraction, resulting in a high-dimensional feature vector. It introduces learnable time-decayed position encoding, sensor confidence scaling, and attention head weighting mechanisms to enhance the modeling ability for temporal dependencies and uncertain perceptions.

[0171] Subsequently, a teacher-student improved SAC network was used to improve the features extracted by the multilayer Transformer encoder. Generate the drone's raw motion based on the state input. Candidate actions are obtained by performing VO security shield correction before execution. Recording during the interaction The quintuples are stored in the experience replay pool. When the replay pool reaches its minimum capacity, each iteration samples a small batch from the experience replay pool for network updates.

[0172] Specifically, in the teacher training phase, a teacher network is trained using the standard SAC algorithm. After training, the teacher model is able to make expert decisions. Once the teacher network has been trained to the point where its performance is stable, its output features are frozen and saved. .

[0173] During the student distillation and joint training phases, the student network continues interactive sampling while utilizing the knowledge of the teacher network to guide the student model. The constraint loss of the student network is obtained by jointly minimizing the following loss. ① Knowledge distillation loss aligned with teacher output ② Reconstruction regularization loss introduced by attention projection low-rank decomposition ③ Directional consistency loss constructed using attention maps ④VO safety trigger loss .

[0174] During joint optimization, the aforementioned confidence weights and time masks are incorporated into feature aggregation and loss weighting to suppress the interference of distant historical and low-confidence observations on learning. Training alternates between "environment steps / gradient update steps" at a fixed ratio, with the target network employing moving average / soft updates until the validation set performance converges. During deployment, only the student network and VO security shield are retained for online inference.

[0175] In this application, the combination of a Transformer feature extraction network and a teacher-student framework-improved SAC network significantly enhances the system's autonomous exploration performance in complex dynamic environments. The performance of the improved teacher-student SAC network in this application and the conventional SAC network during training are compared, and the results are shown in Table 1.

[0176] Table 1 Comparison of Training Results

[0177]

[0178] As shown in Table 1, by improving the SAC network, the number of training convergence rounds was reduced from 763–863 to 557–630, improving training efficiency by approximately 25%; in a dynamic environment of 20×20×3, the average exploration time was shortened from 310.8 seconds to 283.1 seconds, improving task response efficiency by approximately 9%; the average flight path was shortened from 231.9 meters to 223.5 meters, reducing path redundancy and making the trajectory more compact; the number of samples required to achieve an 80% task success rate was reduced from 50 million steps to 10 million steps, improving sample efficiency by 5 times; and the collision rate in a dynamic obstacle environment was reduced from 14%–32% to 8%–13%, significantly improving system safety. Therefore, the navigation method of this invention can significantly improve system safety and navigation efficiency.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A UAV exploration and navigation method based on a sequence enhancement-improved SAC algorithm, characterized in that, Includes the following steps: Step 1: Acquire multimodal sensing data, and form a standardized state vector from the multimodal sensing data at each time step, i.e.: ； in, Indicates time t The state vector, This represents the internal state vector of the UAV; Represents a static obstacle vector. Represents a dynamic obstacle vector. This represents the boundary vector of the unknown region detected by the local map; Step 2: Form a state sequence from the state vectors of each time step. Inputting the improved Transformer network for feature extraction yields high-dimensional feature vectors. The improved Transformer network includes a position encoder, a multi-layer Transformer encoder, a pooling layer, and a concatenation layer. The position encoder encodes the state vector to obtain a sequence matrix. The multi-layer Transformer encoder calculates the time-series hidden representation matrix based on the position-encoded sequence matrix. The pooling layer aggregates the time-series hidden representation matrix. The concatenation layer concatenates the feature representation output from the pooling layer with the state vector output from the position encoder to obtain a high-dimensional feature vector. ; Step 3: Convert the high-dimensional feature vector The input is a teacher-student improved SAC network, which includes a teacher network and a student network. During the training phase, the teacher network is trained using a standard SAC network and provides knowledge distillation signals. The student network inherits the teacher's policy through distillation and combines distillation loss, low-rank reconstruction regularization, attention direction consistency loss, and VO safety triggering loss to form the total loss. During the deployment phase, only the student network is retained for online inference, and the student network outputs the original actions. ; Step 4: Determine if the distance to the nearest obstacle is less than the threshold. If so, perform VO safety shield correction on the original action output by the student network to obtain the corrected action, and then project / clip it to the dynamic feasible set before sending it down for execution. If not, directly execute the original action output by the student network. In step 4, the original action The method for modifying VO Security Shield is as follows: Step 4.1: Determine the speed obstacle set for each obstacle. The calculation formula is as follows: ； in, Represents the set of speed obstacles for the j-th dynamic obstacle; and Let represent the estimated position and velocity of the j-th dynamic obstacle at the current moment, respectively. Indicates time The location of the drone v Indicates candidate velocity, This indicates a time less than the prediction time window, where T represents the prediction time window. Indicates the safety threshold; Step 4.2: Determine the safe speed set based on the speed set of all obstacles. ,in, ; Represents a three-dimensional set. Represents the union; Step 4.3: Obtain the original actions for the teacher-student improved SAC network output. Determine if it belongs to the safe speed set; if so, correct the action. Values If it does not belong to the set of safe speeds, then it is included in the set of safe speeds. Searching for and Minimum distance and belongs to the dynamically feasible set The candidate velocity v is used as the correction action ,Right now: ； in Let v denote the dynamically feasible set, and v denote the safe velocity set. Candidate velocity in the process.

2. The UAV exploration and navigation method based on the sequence enhancement improved SAC algorithm according to claim 1, characterized in that, In step 1, the method for obtaining the boundary vector of the unknown region detected by the local map specifically includes the following steps: Step 1.1: Extract all preface points based on the local 3D occupancy raster map; Step 1.2: For each detected preamble point, calculate the information gain; the calculation formula is: ； in, Indicates the frontier point The corresponding information gain; Indicates the frontier point The corresponding local information gain; and These are the weighting coefficients; Indicates the current position. This indicates the cost or penalty associated with distance; Step 1.3: Use the leading edge point and its corresponding information gain as the boundary vector of the unknown region detected by the local map. .

3. The UAV exploration and navigation method based on the sequence enhancement improved SAC algorithm according to claim 1, characterized in that, In step 2, the displacement encoder processes the state sequence. The method for position encoding is as follows: Calculate the confidence-weighted state embedding vector : ； in, This represents the confidence-weighted state embedding vector. , The parameters represent the linear mapping; This represents the activation / regularization function. This represents a channel-by-channel differentiable scaling function. Indicates element-wise multiplication; Calculate the position-encoded state vector: ； in, This represents a learnable time-decayed positional encoding. Let L represent the confidence-weighted state embedding vector at time ti, and L represent the length of the state sequence.

4. The UAV exploration and navigation method based on the sequence enhancement improved SAC algorithm according to claim 1, characterized in that, In step 2, the improved Transformer network used includes a multi-layer Transformer encoder comprising two stacked Transformer coding layers. Each Transformer coding layer includes an 8-head self-attention mechanism, a feedforward network, a normalization layer, and an output layer connected in sequence.

5. The UAV exploration and navigation method based on the sequence enhancement improved SAC algorithm according to claim 1, characterized in that, In the aforementioned teacher-student improved SAC network, the loss function of the student network is: ； in, , , and These represent knowledge distillation losses, respectively. Refactoring regular expressions Prior loss of attentional direction consistency Loss triggered by VO Security Shield The corresponding weights.

6. The UAV exploration and navigation method based on the sequence enhancement improved SAC algorithm according to claim 5, characterized in that, Refactoring regular expressions The calculation formula is: ； in, The projection matrix of the Transformer encoder is represented by U and V, which are the two matrices decomposed from the projection matrix W. This represents the Frobenius norm.

7. The UAV exploration and navigation method based on the sequence enhancement improved SAC algorithm according to claim 5, characterized in that, VO Security Shield Trigger Loss The calculation formula is: ； in, Indicates the average of experience; This represents the original action by which the teacher and student improve the output of the SAC network. With candidate actions The regularized square, Indicates an indicator function.

8. The UAV exploration and navigation method based on the sequence enhancement improved SAC algorithm according to claim 1, characterized in that, In step 4, the dynamic feasible set The calculation formula is: ； in This is the maximum flight speed of the drone; This is the maximum acceleration; The actual flight speed in the previous time step. Indicates the control cycle.

9. A UAV exploration and navigation method based on a sequence enhancement improved SAC algorithm according to any one of claims 1 to 4, characterized in that, It also includes the following steps: A three-dimensional simulation environment is constructed, in which a UAV is equipped with a high-sensitivity depth camera, a three-dimensional LiDAR, and an inertial measurement unit; Simulations are performed in a 3D simulation environment to acquire multimodal sensing data; Steps 1-4 are performed based on multimodal sensing data to train the improved Transformer network and the teacher-student improved SAC network.

Images