Unmanned aerial vehicle navigation method and system based on vision and reinforcement learning

Abstract

The present application relates to navigation control technical field, especially to a kind of unmanned aerial vehicle navigation method and system based on vision and reinforcement learning, method includes: obtaining lagged first time visual image, real-time second time depth image and the continuous sensing data between them;Through the preset distillation vision network extraction global semantic features and with positive shadow image matching, the visual positioning anchor point of first time is solved;Inertial compensation positioning point of current time is obtained by using sensing data to calculate displacement and attitude change in time window, and mapping visual positioning anchor point to second time;Tensor splicing is carried out to inertial compensation positioning point and depth image to build reinforcement learning state space, and control instruction is generated by inputting strategy network;Through inertial compensation mechanism, the time-space deviation caused by visual calculation delay is eliminated, the flight trajectory oscillation problem caused by different step of multi-source heterogeneous data is solved, and the high robustness autonomous navigation and smooth control of unmanned aerial vehicle in dynamic environment are realized.

Description

Technical Field

[0001] This invention relates to the field of navigation and control technology, and in particular to a navigation method and system for unmanned aerial vehicles (UAVs) based on vision and reinforcement learning. Background Technology

[0002] With the widespread application of drone technology in complex industrial scenarios, how to achieve long-distance autonomous navigation and obstacle avoidance in environments without a Global Navigation Satellite System (GNSS) has become a key technical challenge in the field of robotics. Existing solutions mainly include geometric navigation methods based on Simultaneous Localization and Mapping (SLAM) and end-to-end navigation methods based on Deep Reinforcement Learning (DRL). The former performs global planning by building an environmental map, while the latter directly establishes a mapping relationship between perception and action through neural networks.

[0003] However, in practical applications, the above-mentioned technical solutions still face severe challenges when dealing with highly dynamic flight missions. Although the introduction of large pre-trained visual models (such as the DINO series) can significantly improve the feature matching accuracy and positioning robustness in complex scenarios, the huge amount of computation of such models limits the overall perception frequency of the system. At the same time, the obstacle avoidance and attitude control of UAVs have extremely high requirements for real-time performance.

[0004] The current conventional approach is to directly stitch the visual positioning results with real-time obstacle avoidance data as the status input. However, when UAVs perform high-speed cruise or agile maneuvers to avoid dynamic obstacles, this approach often leads to obvious trajectory oscillations in the flight control system. UAVs struggle to achieve smooth and stable path tracking while maintaining high-speed flight, and often experience lag in control command response or overshoot in order to correct deviations, which seriously affects the safety and efficiency of operations. Summary of the Invention

[0005] The main objective of this invention is to provide a drone navigation method based on vision and reinforcement learning, which aims to solve the problem of poor dynamic environment adaptability of traditional drone navigation technology.

[0006] To achieve the above objectives, the present invention provides a drone navigation method based on vision and reinforcement learning, the method comprising the following steps: The flight status data of the UAV is acquired, including a visual image at a first moment, a depth image at a second moment, and sensor data between the first moment and the second moment, wherein the first moment lags behind the second moment. Based on the preset distillation visual network, feature extraction is performed on the visual image at the first moment to obtain a global semantic feature vector. The global semantic feature vector is then matched with a preset orthophoto to obtain the visual positioning anchor point at the first moment. Based on the sensing data between the first and second moments, the displacement increment and attitude change between the first and second moments are obtained, and the visual positioning anchor point is mapped to the second moment to obtain the inertial compensation positioning point at the current moment. A global navigation state is constructed based on the inertial compensation positioning point and the preset navigation endpoint. The global navigation state is then tensor-stitched with the depth image at the second time step to obtain the current state space for reinforcement learning. The current state space is input into the navigation policy network to obtain navigation control commands for the UAV.

[0007] Optionally, the step of extracting features from the visual image at the first moment according to a preset distillation visual network to obtain a global semantic feature vector includes the following steps: Obtain the weights of the pre-trained visual Transformer model, and perform knowledge distillation on the visual Transformer model based on the task scenario data to obtain the distilled visual network. The visual image at the first moment is input into the distilled visual network, and features with rotation invariance and illumination robustness are extracted as the global semantic feature vector.

[0008] Optionally, matching the global semantic feature vector with a preset orthophoto to obtain the visual positioning anchor point at the first moment includes the following steps: The preset orthophoto image is divided into multiple grid units, and a key feature vector library corresponding to each grid unit is established; Obtain the cosine similarity between the global semantic feature vector and each key feature vector in the key feature vector library; The center geographic coordinates of the grid cell with the highest cosine similarity are selected as the visual positioning anchor point at the first moment, and the cosine similarity is used as the confidence factor of the positioning anchor point.

[0009] Optionally, obtaining the displacement increment and attitude change from the first time moment to the second time moment based on the sensing data from the first time moment to the second time moment includes: Based on the sensor data, obtain the triaxial acceleration and triaxial angular velocity between the first and second moments; The attitude rotation matrix of the UAV is updated based on the three-axis angular velocity, and the three-axis acceleration is projected onto the navigation coordinate system based on the updated attitude rotation matrix; The displacement increment is obtained by performing a second integral on the projected triaxial acceleration. The three-axis angular velocities are integrated to obtain the attitude change.

[0010] Optionally, constructing a global navigation state based on the inertial compensation positioning point and a preset navigation endpoint includes: Obtain the Euclidean distance between the inertial compensation positioning point and the preset navigation endpoint, and use the Euclidean distance as the distance state component; Obtain the angle between the current nose direction of the UAV and the direction vector pointing from the inertial compensation positioning point to the navigation endpoint, and use it as the azimuth state component; The linear velocity and angular velocity of the UAV at the second moment are obtained as dynamic state components; The distance state component, orientation state component, and dynamic state component are combined to generate the global navigation state.

[0011] Optionally, the navigation strategy network is constructed based on a dual-delay depth deterministic policy gradient algorithm, and the navigation strategy network includes an Actor network and a Critic network; The Actor network is used to receive the current state space and output the navigation control commands in the continuous action space; The Critic network is used to evaluate the value of executing the navigation control command in the current state space, so as to guide the parameter update of the Actor network.

[0012] Optionally, during the training of the navigation strategy network, when the confidence factor is lower than a preset threshold, the navigation reward weight guided by the global navigation state is reduced in the training reward function, and the obstacle avoidance reward weight guided by the depth image at the second time step is increased.

[0013] Optionally, before acquiring the UAV's flight status data, the method further includes: The original aerial images of the navigation area are acquired, and a two-dimensional panoramic orthophoto is obtained by image stitching. The two-dimensional panoramic orthophoto is subjected to gridded feature extraction to construct the preset orthophoto and the feature index library of the orthophoto.

[0014] Optionally, inputting the current state space into the navigation policy network to obtain navigation control commands for the UAV includes: Obtain the normalized action value output by the navigation strategy network, wherein the normalized action value is within a preset value range; Based on the maximum linear velocity and maximum angular velocity of the UAV, the normalized motion values ​​are linearly mapped to the target physical control quantities. The target physical control quantity is smoothed and filtered to obtain the navigation control command containing the target linear velocity and the target angular velocity.

[0015] To achieve the above objectives, the present invention also provides a drone navigation system based on vision and reinforcement learning, the system comprising: The data acquisition module is used to acquire flight status data of the UAV. The flight status data includes a visual image at a first moment, a depth image at a second moment, and sensor data between the first moment and the second moment, wherein the first moment lags behind the second moment. A visual positioning module is used to extract features from a visual image at a first moment based on a preset distillation visual network to obtain a global semantic feature vector, and to match the global semantic feature vector with a preset orthophoto to obtain a visual positioning anchor point at the first moment. The state compensation module is used to obtain the displacement increment and attitude change from the first time to the second time based on the sensing data between the first time and the second time, and to map the visual positioning anchor point to the second time to obtain the inertial compensation positioning point at the current time. A state construction module is used to construct a global navigation state based on the inertial compensation positioning point and the preset navigation endpoint, and to perform tensor concatenation between the global navigation state and the depth image at the second time step to obtain the current state space of reinforcement learning. The navigation control module is used to input the current state space into the navigation policy network to obtain navigation control commands for the UAV.

[0016] To achieve the above objectives, the present invention also provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program and the processor executes the computer program to implement the above-described method.

[0017] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, wherein a processor executes the computer program to implement the above-described method.

[0018] The beneficial effects that this invention can achieve are as follows: This invention effectively solves the technical problems of traditional visual navigation schemes when introducing large models for feature extraction, such as delayed perception feedback and temporal misalignment of multi-source data caused by computational time consumption, which lead to trajectory oscillation, path tracking instability and slow obstacle avoidance response in high-speed or high-maneuver flight. By constructing an asynchronous spatiotemporal alignment compensation mechanism based on high-frequency sensor data and combining lightweight visual semantic matching and deep reinforcement learning decision framework based on knowledge distillation, this invention effectively solves the technical problems of traditional visual navigation schemes when introducing large models for feature extraction, such as delayed perception feedback and temporal misalignment of multi-source data, which lead to trajectory oscillation, path tracking instability and slow obstacle avoidance response in high-speed or high-maneuver flight. This invention achieves high-frequency real-time response and precise control in dynamic environments under the condition of limited computing power of airborne edge computing platforms. Specifically, by utilizing the kinematic integral of inertial data to map the lagging visual positioning anchor point to the current moment, the time difference in the "perception-execution" loop is completely eliminated, ensuring the strict spatiotemporal consistency of the input state of the reinforcement learning policy network. This suppresses the step jump of control commands and significantly improves the smoothness of flight trajectory and the level of abrupt change control. At the same time, based on the global semantic feature extraction of the distilled visual network, the system is endowed with strong robustness under extreme conditions such as drastic changes in lighting, sparse texture, or severe tilting of the fuselage. This enables the UAV to balance the accuracy of macroscopic path planning and the agility of microscopic obstacle avoidance in environments without satellite signals, greatly improving the safety threshold and mission efficiency of all-weather autonomous operation. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0020] Figure 1 This is a flowchart illustrating the method in Embodiment 1 of the present invention; Figure 2 This is a structural block diagram of the system in Embodiment 9 of the present invention.

[0021] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention. Example

[0024] Reference Figure 1 This embodiment provides a drone navigation method based on vision and reinforcement learning, the method including the following steps: The flight status data of the UAV is acquired, including a visual image at a first moment, a depth image at a second moment, and sensor data between the first moment and the second moment, wherein the first moment lags behind the second moment. Based on the preset distillation visual network, feature extraction is performed on the visual image at the first moment to obtain a global semantic feature vector. The global semantic feature vector is then matched with a preset orthophoto to obtain the visual positioning anchor point at the first moment. Based on the sensing data between the first and second moments, the displacement increment and attitude change between the first and second moments are obtained, and the visual positioning anchor point is mapped to the second moment to obtain the inertial compensation positioning point at the current moment. A global navigation state is constructed based on the inertial compensation positioning point and the preset navigation endpoint. The global navigation state is then tensor-stitched with the depth image at the second time step to obtain the current state space for reinforcement learning. The current state space is input into the navigation policy network to obtain navigation control commands for the UAV.

[0025] It should be noted that in traditional UAV navigation technology, when performing high-dynamic flight missions in environments without a global satellite navigation system, visual positioning results and real-time obstacle avoidance data are directly spliced ​​together as state inputs, causing trajectory oscillations in the flight control system. Among these issues, control command response lag and overshoot are frequently observed. This phenomenon stems from the contradiction between the limited perception frequency and the real-time control requirements in high-dynamic scenarios. Specifically, it manifests as reduced smoothness and stability during path tracking, which in turn negatively impacts operational safety and mission efficiency.

[0026] To address the aforementioned issues, this embodiment provides a UAV navigation method based on vision and reinforcement learning. Specifically, the UAV needs to continuously acquire its flight status data during flight. This flight status data typically includes a visual image at a first moment, a depth image at a second moment, and sensor data between the first and second moments. For example, the UAV can be equipped with a visual sensor (binocular / monocular camera) to capture visual images, a structured light or time-of-flight (ToF) sensor to acquire depth images, and an inertial measurement unit (IMU) to record accelerometer and gyroscope data. In practice, due to the different working principles of the sensors and data processing delays, there may be a slight time difference between the acquisition time of the visual image (first moment) and the acquisition time of the depth image (second moment), with the first moment slightly lagging behind the second moment. The sensor data continuously records the motion information of the UAV within this time interval.

[0027] To achieve global positioning of the UAV, features need to be extracted from the visual image at the first moment using a pre-defined distillation vision network to obtain a global semantic feature vector. A lightweight convolutional neural network (CNN) is employed as the distillation vision network. This network, after training, can extract distinctive features from the image, such as building outlines and road textures. These features are encoded into a global semantic feature vector. Subsequently, this global semantic feature vector is matched with a pre-defined orthophoto to obtain the visual positioning anchor point at the first moment. The orthophoto can be a pre-created high-resolution satellite image or aerial photograph covering the entire navigation area. By calculating the similarity between the global semantic feature vector and features in different regions of the orthophoto, the corresponding position of the visual image on the orthophoto can be determined; this position is the visual positioning anchor point at the first moment.

[0028] During the visual positioning update interval, the precise position of the UAV needs to be compensated for using inertial measurement. Based on sensor data collected between the first and second time points, the system calculates the displacement increment and attitude change of the UAV during this period. By integrating the three-axis acceleration and three-axis angular velocity output by the inertial measurement unit (IMU), the position change and attitude rotation of the UAV during this period can be estimated. Subsequently, the visual positioning anchor point at the first time point is mapped to the second time point based on the calculated displacement increment and attitude change to obtain the inertial compensated positioning point at the current time. This inertial compensated positioning point combines the global accuracy of visual positioning with the continuity of inertial measurement, providing a smoother and more real-time position estimate.

[0029] To enable the reinforcement learning agent to make effective navigation decisions, a comprehensive state space needs to be constructed. Based on the inertial compensation positioning point and the preset navigation endpoint, a global navigation state can be constructed. By calculating the straight-line distance and direction from the inertial compensation positioning point to the navigation endpoint, and combining this with the UAV's current linear and angular velocities, a global navigation state vector describing the UAV's macroscopic navigation progress is formed. Subsequently, this global navigation state is tensor-concatenated with the depth image at the second time step to obtain the current state space for reinforcement learning. The depth image provides local 3D structural information of the UAV's surrounding environment, which is crucial for obstacle avoidance. By concatenating the global navigation state vector with the pixel data of the depth image, a comprehensive tensor containing global navigation information and local obstacle avoidance information is formed, which serves as the input to the reinforcement learning network.

[0030] Finally, the current state space is input into the navigation policy network to obtain navigation control commands for the UAV. This navigation policy network is a deep neural network trained with reinforcement learning. It receives the concatenated state space as input and outputs a series of values ​​based on its learned policy. These values ​​are interpreted as navigation control commands for the UAV, such as target linear velocity and target angular velocity. These commands are then sent to the UAV's flight control system to guide the UAV toward its navigation endpoint and avoid obstacles. Example

[0031] In this embodiment, the step of extracting features from the visual image at the first moment according to a preset distillation visual network to obtain a global semantic feature vector includes the following steps: Obtain the weights of the pre-trained visual Transformer model, and perform knowledge distillation on the visual Transformer model based on the task scenario data to obtain the distilled visual network. The visual image at the first moment is input into the distilled visual network, and features with rotation invariance and illumination robustness are extracted as the global semantic feature vector.

[0032] It should be noted that, considering the limitations of computing power and power consumption of the UAV's onboard computing platform, directly deploying a large-scale pre-trained model for real-time inference would lead to unacceptable system latency, thereby causing oscillations in navigation and control. Therefore, this embodiment employs knowledge distillation technology to construct a lightweight distilled visual network. This process first involves obtaining the weights of a pre-trained visual Transformer model. These weights are typically derived from large-scale visual Transformer models (such as ViT-Large or ViT-Huge architectures) pre-trained on large-scale public datasets (such as ImageNet or specialized remote sensing datasets). These models possess strong semantic representation capabilities in the feature space and robustness to rotation and illumination changes; in this embodiment, they are defined as the teacher network. Simultaneously, a lightweight convolutional neural network or small Transformer network (such as MobileViT or ResNet-18) with a small number of parameters and fast inference speed is initialized as the student network. To enable the student network to inherit the teacher network's deep understanding of environmental features, especially rotation invariance crucial for orthophoto matching, it is necessary to distill the student network using task scenario data (i.e., image samples collected from the actual UAV operating environment).

[0033] It should also be noted that during the distillation training phase, the core objective is to constrain the feature vectors output by the student network to maintain as much consistency as possible with the feature vectors output by the teacher network in terms of direction. This is because in the navigation logic of this invention, subsequent localization matching is based on vector cosine similarity. The direction of the vector determines semantic consistency, while the magnitude of the vector is usually affected by illumination intensity or contrast. To quantify the effect of this feature alignment and guide the update of network parameters, this embodiment designs a distillation loss function based on cosine embedding. The mathematical expression of this loss function is as follows: ; In the formula, The total distillation loss value calculated in a single iteration is used to update the network gradient through the backpropagation algorithm; N is the size of the training batch, which is the number of input image samples in one iteration. The specific source of this data is the preprocessor loader of the training data. k is the sample index variable, with a value ranging from 1 to N; The k-th input visual image in the current batch is derived from a pre-collected task scenario dataset; Let be the forward propagation function of the student network to be trained, and let the image be the input. The parameters are the weights to be optimized. The corresponding output results This represents the feature vector extracted from the student network; For the forward propagation function of the teacher network with frozen weights, Similarly, weights The standard feature vectors generated for the teacher network, where the symbol · represents the dot product operation of the vectors. Let be the Euclidean norm of the vector.

[0034] Understandably, the above expression physically calculates the cosine similarity between the student's feature vector and the teacher's feature vector, with a value ranging from -1 to +1. The reason for subtracting this similarity from 1 is that when the two vectors are completely aligned, the similarity is 1, and the loss function value is 0, representing optimal learning performance. Conversely, if the directions diverge, the loss value increases, thus penalizing the student network's incorrect representations. This design forces the lightweight student network to ignore high-frequency noise in the image and focus on learning the macroscopic semantic structure that the teacher network is interested in, thereby achieving robustness to illumination and rotation disturbances.

[0035] It is also understandable that after completing the above distillation training, the final distilled visual network is obtained. During the actual navigation mission of the UAV, the system inputs the visual images acquired at the first moment into this distilled visual network. At this time, the network performs forward inference operations, mapping the high-dimensional pixel matrix into a low-dimensional dense vector, which is the global semantic feature vector. This feature vector not only has the characteristics of low-dimensional compactness to support fast retrieval, but also retains geometric invariance to the environmental perspective. It can ensure that even if the UAV's flight attitude is tilted or the ambient lighting is inconsistent with the base map, the generated feature vector can still maintain a small angular distance from the pre-stored orthophoto base map features in the vector space, thus providing a reliable data foundation for subsequent accurate positioning anchor point calculation.

[0036] Example 3: In this embodiment, matching the global semantic feature vector with a preset orthophoto to obtain the visual positioning anchor point at the first moment includes the following steps: The preset orthophoto image is divided into multiple grid units, and a key feature vector library corresponding to each grid unit is established; Obtain the cosine similarity between the global semantic feature vector and each key feature vector in the key feature vector library; The center geographic coordinates of the grid cell with the highest cosine similarity are selected as the visual positioning anchor point at the first moment, and the cosine similarity is used as the confidence factor of the positioning anchor point.

[0037] In this embodiment, the navigation strategy network is constructed based on the dual-delay depth deterministic strategy gradient algorithm, and the navigation strategy network includes an Actor network and a Critic network; The Actor network is used to receive the current state space and output the navigation control commands in the continuous action space; The Critic network is used to evaluate the value of executing the navigation control command in the current state space, so as to guide the parameter update of the Actor network.

[0038] It should be noted that, in this embodiment, for the visual positioning anchor point acquisition step, the system first needs to complete the structured processing and feature library construction of the preset orthophoto (ODM) in the offline stage. The orthophoto is usually a high-resolution, topographically distortion-free two-dimensional geographic base map covering the entire task area, and its pixel coordinates have a strict linear mapping relationship with the real geographic latitude and longitude coordinates. In order to achieve efficient retrieval and matching, pixel-level search cannot be performed directly on the original large-size base map. Instead, the orthophoto needs to be spatially discretized, that is, divided into multiple non-overlapping or slightly overlapping grid units according to the preset physical size (e.g., each grid represents 10 meters × 10 meters of ground). For each grid unit, forward inference is performed using a distilled visual network that is exactly the same as the airborne end or aligned with the feature space to extract the deep semantic features of the grid image block, which is defined as a key feature vector. A one-to-one index relationship is established between this vector and the central geographic coordinates of the grid unit, and stored in the airborne high-dimensional vector database (such as the FAISS library or KD-Tree structure), thereby constructing the key feature vector library.

[0039] During the online operation phase of the UAV mission, after the onboard vision system extracts the global semantic feature vector of the first-moment visual image, the system immediately initiates the feature matching process. The core of this process lies in quantifying the similarity between the semantic features of the current viewpoint (i.e., the query vector) and the features (i.e., the key vectors) of each grid cell in the base database. Cosine similarity is chosen as the sole metric because the feature distribution of visual Transformer models such as DINOv3 is primarily reflected in the directional consistency of the high-dimensional sphere, rather than the magnitude of the vector. The specific similarity calculation process follows the mathematical model below: ; In the formula, This represents the cosine similarity score between the current visual image and the i-th grid cell. Its value is usually between -1 and 1, but after normalization training in the feature space, the effective matching value is usually close to 1. The symbol i is the index number of the grid cell, and its value traverses the entire key feature vector library.

[0040] The global semantic feature vector is output in real time by the airborne distillation vision network, which is derived from the visual image acquired at the first moment and the result of neural network inference. The key feature vector representing the i-th grid cell retrieved from the database originates from the offline preprocessing of the orthophoto gridding.

[0041] In the above expression, by dividing the dot product by the product of the magnitudes of the two vectors, the formula physically eliminates the interference of differences in illumination intensity (usually manifested as scaling of feature vector magnitudes) on the matching results, making the similarity depend only on the cosine of the angle between the two vectors in the feature space. This means that even if the drone flies at dusk or on a cloudy day (weak illumination, smaller vector magnitudes), as long as the semantic structure of the ground (such as the shape of a road intersection) matches the bottom... Figure 1 If the vectors are aligned (in the same direction), the calculated similarity will remain high, thus ensuring the robustness of the localization.

[0042] After calculating the similarity set between the global semantic feature vector and all (or coarsely screened local region) key feature vectors in the database, the system performs a maximum value filtering operation. The system selects the grid cell index with the largest value in the set and extracts the pre-stored center geographic coordinates of that grid cell as the visual positioning anchor point for the first moment. This anchor point provides the most likely absolute latitude and longitude position of the UAV at the first moment. More importantly, this maximum similarity value is directly reused by the system as the confidence factor for the positioning anchor point.

[0043] Understandably, this factor can be interpreted as the system's confidence level in the current positioning result. If the value is high (e.g., greater than 0.85), it indicates that the current viewpoint highly matches a certain point on the base map, and the positioning is very reliable; conversely, if the value is very low (e.g., less than 0.4), it suggests that the drone may be in an area obscured by clouds, with a severe attitude tilt, or not covered by the base map, in which case the visual positioning anchor point may have a large error. This confidence factor will then be passed to the subsequent navigation policy network to adjust the weight of the reinforcement learning algorithm in accepting visual navigation data, thereby achieving adaptive perception of environmental uncertainty at the algorithm level.

[0044] Example 4: In this embodiment, obtaining the displacement increment and attitude change from the first time moment to the second time moment based on the sensing data from the first time moment to the second time moment includes: Based on the sensor data, obtain the triaxial acceleration and triaxial angular velocity between the first and second moments; The attitude rotation matrix of the UAV is updated based on the three-axis angular velocity, and the three-axis acceleration is projected onto the navigation coordinate system based on the updated attitude rotation matrix; The displacement increment is obtained by performing a second integral on the projected triaxial acceleration. The three-axis angular velocities are integrated to obtain the attitude change.

[0045] It should be noted that due to the time-consuming inference of deep learning models, by the time the visual positioning anchor point calculation is completed, the UAV has already undergone displacement and attitude changes in physical space. To eliminate this error, the system needs to call the state compensation module to perform dead reckoning based on inertial navigation. First, the system accesses the circular buffer of the onboard flight control computer and, based on the recorded timestamp index, accurately extracts the sensor data within a specific time window from the first moment t1 to the second moment t2. This sensor data mainly includes high-frequency three-axis acceleration data and three-axis angular velocity data output by the microelectromechanical system (MEMS) inertial measurement unit. Among them, the three-axis acceleration data reflects the specific force change of the UAV in the body coordinate system, while the three-axis angular velocity data reflects the angular rate of the UAV's rotation around the three axes of the body.

[0046] After acquiring the basic data, the system first performs an update operation on the attitude rotation matrix. This is because the inertial sensor is fixed to the UAV body, and its measurement data is based on the body coordinate system, which is subject to tumbling. Navigation and positioning, on the other hand, are based on a fixed navigation coordinate system (usually the ENU, North-East-South coordinate system). In order to convert the body motion into navigation space, an attitude rotation matrix describing the rotational relationship between the body and the navigation system must be maintained in real time. The system uses the three-axis angular velocity data within the time window to update the initial attitude matrix at the first moment by integrating it using quaternion differential equations or the Rodrigues formula, thus obtaining the instantaneous attitude rotation matrix R(t) at any moment within that time period.

[0047] Subsequently, the system performs acceleration projection and gravity compensation operations. Since the accelerometer measures specific force that includes a component of gravitational acceleration, direct integration would lead to a large error in the vertical direction. Therefore, the system uses the updated attitude rotation matrix R(t) to project the original three-axis acceleration in the body coordinate system to the navigation coordinate system and subtracts the local gravitational acceleration constant. The mathematical expression of this physical process is as follows: ; In the formula, Let be the actual acceleration of motion in the navigation coordinate system at time t; R(t) is the attitude rotation matrix at time t, and its data comes from the integral calculation based on the gyroscope angular velocity data; The original sampled value output by the accelerometer at time t is directly derived from the airborne IMU sensor; This is the vector of gravitational acceleration.

[0048] After obtaining the net acceleration under the navigation system, the system performs a core quadratic integral operation to obtain the displacement increment. According to Newton's laws of kinematics, displacement is the double integral of acceleration with respect to time. In a discrete digital system, this process is achieved through summation. The mathematical model for calculating the displacement increment from the first moment to the second moment is as follows: ; In the formula, This is the displacement increment generated by the UAV within the lag time window, which is the correction vector that needs to be superimposed on the visual positioning anchor point. The initial instantaneous velocity of the drone at the first moment; The time window represents the length of the time window, i.e. the system latency caused by visual processing, where t1 represents the first moment and t2 represents the second moment. The time differential operator representing a double integral can be understood as the differential operator for the acceleration term. Two integration operations are performed consecutively within the time range from t1 to t2.

[0049] Understandably, the double integral term in the above expression (which is represented by double accumulation in the discrete domain) aims to deduce the change in velocity through acceleration, and then deduce the change in position. In turn, by using inertial data with a high sampling rate (usually greater than 200Hz), the motion trajectory of the UAV can be reproduced with extremely high accuracy in a short period of time (usually less than 100ms) when external observation is lacking.

[0050] Simultaneously, the system performs single-integral calculations on the three-axis angular velocities to directly obtain the attitude change, typically represented as a quaternion increment or Euler angle increment. This change is used to correct the nose-up orientation from the first moment to the actual orientation at the current moment. Finally, the system superimposes the calculated displacement increment vector onto the geographic coordinates of the acquired visual positioning anchor point and applies the attitude change to the direction vector, thereby generating the inertial compensation positioning point for the current moment. This positioning point is strictly aligned with the depth image at the second moment in the time domain, providing a rigorous physical foundation for subsequently constructing a time-bias-free reinforcement learning state space.

[0051] Example 5: In this embodiment, constructing the global navigation state based on the inertial compensation positioning point and the preset navigation endpoint includes: Obtain the Euclidean distance between the inertial compensation positioning point and the preset navigation endpoint, and use the Euclidean distance as the distance state component; Obtain the angle between the current nose direction of the UAV and the direction vector pointing from the inertial compensation positioning point to the navigation endpoint, and use it as the azimuth state component; The linear velocity and angular velocity of the UAV at the second moment are obtained as dynamic state components; The distance state component, orientation state component, and dynamic state component are combined to generate the global navigation state.

[0052] In this embodiment, the core task of the system is to transform the absolute geographical coordinates of the current inertial compensation positioning point into a relative geometric relationship with respect to the preset navigation endpoint, and to generate a state vector that meets the requirements of the Markov decision process by combining the motion state of the UAV. This is because reinforcement learning policy networks usually have difficulty directly processing absolute latitude and longitude coordinates with huge numerical ranges, while the relative distance and relative angle after normalization can significantly improve the convergence speed and generalization ability of network training.

[0053] Specifically, first, the system calculates the distance-state component. The system reads the coordinates of the inertial compensation positioning point at the current moment, denoted as... And the pre-defined navigation endpoint coordinates, denoted as Both coordinates are based on a unified navigation coordinate system. The system uses the Euclidean distance formula to calculate the straight-line distance between the two points, which serves as a scalar signal to guide the UAV to approach the target. The mathematical model for this calculation process is shown below: ; In the formula, Let t be the Euclidean distance between the UAV and the target at the current time t, i.e., the distance state component; The planar coordinate components of the inertial compensation positioning point at the current moment are derived from the output of the state compensation module after superimposing the displacement increment; The planar coordinate components of the navigation endpoint are derived from preset values ​​set during the mission planning phase. The distance value in the above expression directly represents the remaining flight cost. In the reward function of reinforcement learning, a decrease in this value will generate positive incentives, guiding the agent to continuously shorten the distance to the target. To adapt to the input range of the neural network, this distance... Typically, the value is also normalized by dividing by a preset maximum task distance constant, so that its value is distributed in the range [0,1].

[0054] Secondly, the system performs the calculation of the orientation state component. Distance information alone is insufficient to guide the flight direction; the system must calculate the deviation between the current nose-up orientation and the ideal target direction. The system first calculates based on... and The ideal azimuth angle pointing towards the target is calculated, and then the azimuth deviation is calculated by combining it with the UAV's own yaw angle. The mathematical expression of this physical process is as follows: ; In the formula, This refers to the azimuth deviation, i.e., the azimuth state component. This is a two-parameter arctangent function used to calculate the global azimuth angle when rotating from the east (X-axis) direction to the target vector. Its range is... .

[0055] This represents the actual nose-up angle of the drone at the second moment (i.e., the current moment).

[0056] Secondly, the system acquires dynamic state components. Since the UAV is a physical system with inertial and nonholonomic constraints, its current motion state determines its reachable position at the next moment. The system directly extracts the linear velocity and angular velocity at the second moment from the flight control sensor data. The reason for introducing these two variables as state components is that the reinforcement learning network needs to perceive the current momentum to avoid overshooting the target due to insufficient braking distance during high-speed flight or causing severe oscillations during sharp turns. For example, when the linear velocity is high and the UAV is close to the target, the network should learn to output deceleration commands.

[0057] Finally, the system performs a state combination operation. The normalized distance, azimuth deviation, and dynamic parameters calculated above are concatenated into vectors according to a predetermined dimensional order to generate a low-dimensional dense vector, i.e., the global navigation state. This state vector is then further concatenated with the processed depth image (local perception information) to form a complete reinforcement learning current state space.

[0058] Example 6: In this embodiment, during the training process of the navigation policy network, when the confidence factor is lower than a preset threshold, the navigation reward weight guided by the global navigation state is reduced in the training reward function, and the obstacle avoidance reward weight guided by the depth image at the second time step is increased.

[0059] Understandably, in traditional reinforcement learning navigation tasks, the reward function is typically a linear superposition of a navigation guidance term (encouraging approach to the target) and an obstacle avoidance survival term (penalizing collisions) in a fixed ratio. However, this embodiment considers that visual localization may fail to match under extreme lighting, texture loss, or violent maneuvers. If the agent continues to execute incorrect global navigation commands in such situations, it is highly likely to cause flight instability or collisions. Therefore, this embodiment designs a composite reward function with perceptual adaptive capabilities, which adjusts the allocation of various weights in real time based on the localization anchor point confidence factor.

[0060] It is also understandable that the mathematical model of the total reward function satisfies the expression: ; In the formula, The total instant reward gained at the current time step; This is the global navigation reward item, whose calculation is typically based on the constructed global navigation state. The specific expression for this item is as follows: ; In the formula, , These represent the Euclidean distances between the drone and the preset navigation endpoint at the current and previous moments, respectively. If the difference is positive, a positive reward is given. , All are preset constant coefficients; This is a survival bonus item for obstacle avoidance. The smoothness bonus is defined as a negative penalty for the second derivative (i.e., jerkiness) of the current output control command (linear velocity and angular velocity). It is designed to prevent the output of control signals that jitter violently, protect the onboard motors, and maintain flight stability. The confidence factor objectively reflects the reliability of the current visual positioning. The dynamic weight adjustment function satisfies the expression: ; The preset confidence threshold is used. When the real-time confidence level is higher than this threshold, it indicates good visual matching and accurate localization. The function value approaches 1, and at this point, the total reward function is mainly composed of... The agent actively searches for targets; when the value falls below a certain threshold, it indicates that visual localization may have failed (e.g., the base map is lost), and the function value rapidly decays, approaching 0, leading to... The weight is reduced, making To occupy a dominant position; The symbol β is a hyperparameter that adjusts the steepness of the curve and determines the sensitivity of the switching process.

[0061] e is the natural constant, also known as the Euler number.

[0062] Based on the above expression, during training, when visual positioning becomes unreliable in simulated or real-world scenarios, this mechanism focuses all its attention on the local obstacle information transmitted back by the depth camera (i.e., amplifying obstacle avoidance rewards), prioritizing ensuring that the drone does not collide and maintains a hovering or low-speed exploration state until it flies into a textured area and regains high-confidence positioning. This biomimetic attention transfer mechanism significantly improves the robustness and safety of the navigation system in dynamic and complex environments.

[0063] Example 7: In this embodiment, before acquiring the drone's flight status data, the method further includes: The original aerial images of the navigation area are acquired, and a two-dimensional panoramic orthophoto is obtained by image stitching. The two-dimensional panoramic orthophoto is subjected to gridded feature extraction to construct the preset orthophoto and the feature index library of the orthophoto.

[0064] It should be noted that the system performs the acquisition of original aerial images and the generation of two-dimensional panoramic orthophotos. During the mission planning phase, the system acquires original RGB aerial image sequences covering the entire target navigation area through high-altitude reconnaissance UAVs or satellite remote sensing. Since the original photos have perspective distortion and inconsistent scales, they cannot be directly used for precise measurement. Therefore, the system uses the motion reconstruction structure algorithm and multi-view stereo vision technology from photogrammetry to perform geometric correction and seamless stitching on the sequence images, generating a two-dimensional panoramic orthophoto with a unified map projection coordinate system and eliminating projection differences caused by terrain undulations. Each pixel in this image has precise and unique geographic plane coordinates, thus providing a spatial reference at the meter or even sub-meter level for subsequent navigation.

[0065] Subsequently, the system performs gridded feature extraction and index construction. Since the entire orthophoto image is extremely large, a direct full-image search is computationally infeasible; therefore, it needs to be discretized into structured grid units. The system divides the orthophoto image into a set of M rows and N columns of grid image blocks according to a preset physical scale (e.g., a 10m x 10m area on the ground). For each grid image block, the system calls a distilled visual network for forward inference to extract the global semantic feature vector of that block and calculate the physical coordinates of the grid center. Finally, the system constructs a feature index containing all grid information, the mathematical set of which is defined as follows: ; In the formula, The completed orthophoto feature index library is a structured dataset stored in airborne non-volatile memory. The symbol K represents the total number of grid cells, which depends on the total area of ​​the navigation area and the granularity of individual grids. These are binary elements, which constitute the basic storage unit of the database. This represents the key feature vector of the i-th grid cell. The vector is obtained by cropping the local image patch with index i in the orthophoto and inputting it into a pre-trained and distilled visual Transformer network to extract the output tensor. To ensure the accuracy of online matching, the network weights used here must be strictly consistent with the network weights used in real-time navigation on the airborne end. Physically, this vector is a compressed encoding of the semantic information of the plot, such as texture, shape, and road topology.

[0066] This represents the center geographic coordinates of the i-th grid cell, typically expressed as a two-dimensional vector. This coordinate data is obtained based on the geometric rules governing grid division, and its calculation formula is as follows: ; The absolute geographic coordinates of the top left vertex of the orthophoto; , Let be the column index and row index of the i-th grid in the image matrix, respectively. The actual physical side length of the ground represented by a single grid (in meters); This is the map projection rotation matrix, used to handle the angle between the image coordinate system and the navigation coordinate system.

[0067] Understandably, the reason for building the library using the above expressions is to establish a deterministic lookup table. When the vision system finds that the current viewpoint is similar to the i-th feature in the library during online flight, the system can directly look up the table to obtain the corresponding coordinates as the positioning anchor point without performing complex geometric calculations. This architecture design of heavy offline computation and light online retrieval greatly reduces the real-time computing load on the airborne end and ensures the real-time response of the navigation system under high-speed flight.

[0068] Example 8: In this embodiment, inputting the current state space into the navigation policy network to obtain navigation control commands for the UAV includes: Obtain the normalized action value output by the navigation strategy network, wherein the normalized action value is within a preset value range; Based on the maximum linear velocity and maximum angular velocity of the UAV, the normalized motion values ​​are linearly mapped to the target physical control quantities. The target physical control quantity is smoothed and filtered to obtain the navigation control command containing the target linear velocity and the target angular velocity.

[0069] In this embodiment, the onboard navigation strategy network (i.e., the trained TD3Actor network) has completed feature extraction and inference of the current state space and generated a multi-dimensional action vector in the output layer. Because deep reinforcement learning networks typically use the hyperbolic tangent function as the activation function in the output layer to ensure gradient propagation stability during training, the normalized action value directly output by the network is strictly limited to a preset numerical range [-1, 1]. This value only represents the relative intensity of the action and has no specific physical dimensions, therefore it cannot directly drive the UAV motors.

[0070] Therefore, the system first obtains the normalized action value, denoted as a vector. .in, The normalized output representing the linear velocity control dimension. The normalized output representing the angular velocity control dimension, and the data source for both is the forward inference result of the policy network.

[0071] Subsequently, the system performs a linear mapping operation from the normalized domain to the physical domain. The UAV's physical motion capabilities are limited by its airframe structure and propulsion system, and it has a definite maximum flight linear velocity. With maximum flight angular velocity These two parameters are typically preset during system initialization based on the UAV's hardware manual or safe flight envelope. To transform the abstract neural network output into target physical control quantities with actual physical meaning, the system employs the following linear transformation formula: ; In the formula, This represents the desired target linear velocity at the current moment (unit: meters per second). Since UAVs typically only fly forward and not backward in routine navigation missions, the interval [-1,1] needs to be mapped to [0,...]. ] Term in the formula The original motion value is shifted and scaled to the [0,1] range, then multiplied by Complete the dimensional restoration.

[0072] symbol This represents the desired target angular velocity at the current moment (unit: radians / second). Through this parameterized mapping mechanism, the same reinforcement learning model can be deployed on drone platforms with different dynamic performance characteristics, requiring only minor modifications. and Two parameters are sufficient; there is no need to retrain the network.

[0073] Finally, the system performs smoothing filtering on the target physical control input. Because reinforcement learning networks are extremely sensitive to minute changes in state input, the directly output control signal often contains high-frequency noise or abrupt step changes (e.g., a sudden stop from full speed). Sending such signals directly to the underlying flight controller would cause drastic fluctuations in motor speed, increasing mechanical wear and even triggering airframe vibration. Therefore, this embodiment introduces a first-order exponential moving average filter to generate the final navigation control command. The mathematical model for this smoothing process is as follows: ; In the formula, This indicates the final smooth control commands sent to the flight control system (corresponding to linear velocity and angular velocity, respectively). symbol This refers to the target physical control quantity calculated in the previous step; symbol This is the final instruction value from the previous control cycle, and the data comes from the historical records in the system memory; The symbol γ is the smoothing coefficient, and its value is usually between (0,1) (e.g., 0.1 to 0.3).

[0074] The current control commands are not entirely determined by the current neural network output, but rather retain most of the motion trend from the previous moment, incorporating only a small portion of new changes. The reason for using the above expression is that it is mathematically equivalent to a low-pass filter, which can effectively filter out high-frequency jitter components in the control signal, ensuring that the drone's acceleration, deceleration, and turning actions are smooth and fluid, thereby improving flight stability and video smoothness.

[0075] Example 9: As attached Figure 2 As shown, this embodiment provides a drone navigation system based on vision and reinforcement learning, the system comprising: The data acquisition module is used to acquire flight status data of the UAV. The flight status data includes a visual image at a first moment, a depth image at a second moment, and sensor data between the first moment and the second moment, wherein the first moment lags behind the second moment. A visual positioning module is used to extract features from a visual image at a first moment based on a preset distillation visual network to obtain a global semantic feature vector, and to match the global semantic feature vector with a preset orthophoto to obtain a visual positioning anchor point at the first moment. The state compensation module is used to obtain the displacement increment and attitude change from the first time to the second time based on the sensing data between the first time and the second time, and to map the visual positioning anchor point to the second time to obtain the inertial compensation positioning point at the current time. A state construction module is used to construct a global navigation state based on the inertial compensation positioning point and the preset navigation endpoint, and to perform tensor concatenation between the global navigation state and the depth image at the second time step to obtain the current state space of reinforcement learning. The navigation control module is used to input the current state space into the navigation policy network to obtain navigation control commands for the UAV.

[0076] This embodiment constructs an integrated high-performance UAV navigation system. The system adopts a layered and decoupled modular architecture design, which aims to solve the problem of multi-source heterogeneous data fusion and real-time decision-making under the limited computing resources of the edge side. Logically, the system is mainly divided into a data acquisition module, a visual positioning module, a state compensation module, a state construction module, and a navigation control module. Each module realizes high-speed data bus communication through shared memory or message queue mechanism, and together they form a closed-loop perception-decision-control loop.

[0077] The system's front end consists of a data acquisition module, which acts as the sensory nerve center of the entire navigation system, responsible for parallel management and synchronization of raw data streams from multiple sensors. During operation, this module establishes a precise global time base and performs sampling control on the vision sensor, depth sensor, and inertial measurement unit. Specifically, the module captures and marks two key time points: the first moment, corresponding to the instant the monocular visual image is exposed (this data is defined as a lagging node on the timeline due to the high time consumption of subsequent processing); and the second moment, corresponding to the latest system clock, at which time a low-latency real-time depth image is acquired. Simultaneously, the module continuously records high-frequency inertial sensor data streams between the first and second moments, including three-axis acceleration and three-axis angular velocity, and establishes a strict indexed association between this continuous time series data and the visual image, thus providing complete data support for subsequent elimination of the time lag caused by visual processing.

[0078] The visual positioning module receives the first-moment visual image output from the data acquisition module and primarily undertakes high-level semantic understanding and global positioning tasks. Internally, this module deploys a lightweight visual neural network processed by knowledge distillation, capable of deep feature extraction from the input visual image, generating a global semantic feature vector with rotation invariance and illumination robustness. Subsequently, the module performs content-based image retrieval, comparing the extracted real-time feature vector with a pre-built orthophoto feature library in onboard memory. By calculating the directional similarity between feature vectors, the module can quickly locate the best-matching grid region on a large geographic base map and establish the geographic coordinates of the grid center as the UAV's visual positioning anchor point at the first moment. Although this anchor point has high absolute geographic accuracy, its direct use, reflecting the UAV's position at a past moment, can lead to navigation control lag and oscillations.

[0079] To correct this time-domain deviation, the data stream enters the state compensation module. This module is a crucial bridge connecting perception lag and real-time control, and its core function is to perform short-term dead reckoning based on inertial navigation. The module reads inertial sensor data between the first and second moments and reconstructs the UAV's flight dynamics using kinematic principles. Specifically, the module first updates the UAV's attitude matrix using angular velocity data, projects the acceleration in the body coordinate system to the navigation coordinate system and removes the gravity component, and then performs a second integral operation on the net acceleration over time. Through this process, the module accurately calculates the relative displacement increment and attitude change of the UAV during the visual processing window and superimposes these increments onto the outdated visual positioning anchor point. After compensation processing, the module outputs the inertial compensation positioning point at the current moment, whose coordinates are now perfectly aligned with the real-time depth image at the second moment, thus eliminating the time asynchrony between heterogeneous sensors.

[0080] Subsequently, the state construction module is responsible for transforming the geographical information of the physical space into the abstract state space required by the reinforcement learning algorithm. This module receives the coordinates of the inertial compensation positioning point and the preset navigation endpoint. First, it calculates the Euclidean distance and azimuth deviation between the two in a Cartesian coordinate system, transforming the absolute geographical location into relative navigation guidance features. To endow the system with obstacle avoidance capabilities, this module simultaneously receives the real-time depth image at the second time step, downsampling it to extract the spatial distribution features of local obstacles. In addition, the module obtains the current linear velocity and angular velocity from the flight control layer as dynamic constraint information. Finally, the module performs tensor splicing operations in the data channel dimension, fusing the navigation guidance features, local obstacle avoidance features, and dynamic features into a high-dimensional multimodal state tensor, forming the input foundation of the reinforcement learning policy network.

[0081] Finally, the navigation control module, acting as the system's decision-making and execution terminal, generates specific flight commands based on the input state tensor. Internally, this module runs a trained navigation strategy network that directly outputs normalized action values ​​through forward inference. To transform abstract values ​​into physical control signals, the module first performs a linear mapping based on the UAV's maximum flight speed limit, calculating the target linear velocity and target angular velocity. Given the potential for transient jitter in the neural network output, the module further introduces a smoothing filtering mechanism with historical memory characteristics to perform low-pass filtering on the target control quantity, preserving the main trend of the control quantity from the previous moment and integrating incremental information from the current inference value. The smoothed navigation control commands are ultimately sent to the UAV's underlying flight controller, driving the motors to perform robust and smooth maneuvering flight, thereby achieving autonomous navigation in complex dynamic environments.

[0082] Furthermore, in one embodiment, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0083] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a device including one or any combination of the above-mentioned memories. The computer may be a variety of computing devices, including smart terminals and servers.

[0084] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0085] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0086] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0087] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a multimedia terminal device (which may be a mobile phone, computer, television receiver, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0089] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A drone navigation method based on vision and reinforcement learning, characterized in that, The method includes the following steps: The flight status data of the UAV is acquired, including a visual image at a first moment, a depth image at a second moment, and sensor data between the first moment and the second moment, wherein the first moment lags behind the second moment. The visual image at the first moment is extracted according to the preset distillation visual network to obtain a global semantic feature vector. The global semantic feature vector is then matched with the preset orthophoto to obtain the visual positioning anchor point at the first moment. Based on the sensing data between the first and second moments, the displacement increment and attitude change between the first and second moments are obtained, and the visual positioning anchor point is mapped to the second moment to obtain the inertial compensation positioning point at the current moment. A global navigation state is constructed based on the inertial compensation positioning point and the preset navigation endpoint. The global navigation state is then tensor-stitched with the depth image at the second time step to obtain the current state space for reinforcement learning. The construction of the global navigation state based on the inertial compensation positioning point and the preset navigation endpoint includes: obtaining the Euclidean distance between the inertial compensation positioning point and the preset navigation endpoint, and using the Euclidean distance as the distance state component; obtaining the angle between the current nose-up orientation of the UAV and the direction vector pointing from the inertial compensation positioning point to the navigation endpoint, and using this angle as the azimuth state component; obtaining the linear velocity and angular velocity of the UAV at the second time step, and using these as dynamic state components; and combining the distance state component, the azimuth state component, and the dynamic state component to generate the global navigation state. The current state space is input into the navigation policy network to obtain navigation control commands for the UAV.

2. The UAV navigation method based on vision and reinforcement learning as described in claim 1, characterized in that, The step of extracting features from the visual image at the first moment according to a preset distillation visual network to obtain a global semantic feature vector includes the following steps: Obtain the weights of the pre-trained visual Transformer model, and perform knowledge distillation on the visual Transformer model based on the task scenario data to obtain the distilled visual network. The visual image at the first moment is input into the distilled visual network, and features with rotation invariance and illumination robustness are extracted as the global semantic feature vector.

3. The UAV navigation method based on vision and reinforcement learning as described in claim 1, characterized in that, The step of matching the global semantic feature vector with a preset orthophoto to obtain the visual positioning anchor point at the first moment includes the following steps: The preset orthophoto image is divided into multiple grid units, and a key feature vector library corresponding to each grid unit is established; Obtain the cosine similarity between the global semantic feature vector and each key feature vector in the key feature vector library; The center geographic coordinates of the grid cell with the highest cosine similarity are selected as the visual positioning anchor point at the first moment, and the cosine similarity is used as the confidence factor of the positioning anchor point.

4. The UAV navigation method based on vision and reinforcement learning as described in claim 1, characterized in that, The step of obtaining the displacement increment and attitude change from the first time moment to the second time moment based on the sensing data from the first time moment to the second time moment includes: Based on the sensor data, obtain the triaxial acceleration and triaxial angular velocity between the first and second moments; The attitude rotation matrix of the UAV is updated based on the three-axis angular velocity, and the three-axis acceleration is projected onto the navigation coordinate system based on the updated attitude rotation matrix; The displacement increment is obtained by performing a second integral on the projected triaxial acceleration. The three-axis angular velocities are integrated to obtain the attitude change.

5. The UAV navigation method based on vision and reinforcement learning as described in claim 3, characterized in that, The navigation strategy network is constructed based on the dual-delay depth deterministic strategy gradient algorithm, and the navigation strategy network includes an Actor network and a Critic network. The Actor network is used to receive the current state space and output the navigation control commands in the continuous action space; The Critic network is used to evaluate the value of executing the navigation control command in the current state space, so as to guide the parameter update of the Actor network.

6. A UAV navigation method based on vision and reinforcement learning as described in claim 3 or 5, characterized in that, During the training of the navigation strategy network, when the confidence factor is lower than a preset threshold, the navigation reward weight guided by the global navigation state is reduced in the training reward function, and the obstacle avoidance reward weight guided by the depth image at the second time step is increased.

7. The UAV navigation method based on vision and reinforcement learning as described in claim 1, characterized in that, Before acquiring the drone's flight status data, the following is also included: The original aerial images of the navigation area are acquired, and a two-dimensional panoramic orthophoto is obtained by image stitching; the gridded features of the two-dimensional panoramic orthophoto are extracted to construct the preset orthophoto and the feature index library of the orthophoto.

8. The UAV navigation method based on vision and reinforcement learning as described in claim 1, characterized in that, The step of inputting the current state space into the navigation policy network to obtain navigation control commands for the UAV includes: Obtain the normalized action value output by the navigation strategy network, wherein the normalized action value is within a preset value range; Based on the maximum linear velocity and maximum angular velocity of the UAV, the normalized motion values ​​are linearly mapped to the target physical control quantities. The target physical control quantity is smoothed and filtered to obtain the navigation control command containing the target linear velocity and the target angular velocity.

9. A navigation system, characterized in that, The system includes: The data acquisition module is used to acquire flight status data of the UAV. The flight status data includes a visual image at a first moment, a depth image at a second moment, and sensor data between the first moment and the second moment, wherein the first moment lags behind the second moment. A visual positioning module is used to extract features from a visual image at a first moment based on a preset distillation visual network to obtain a global semantic feature vector, and to match the global semantic feature vector with a preset orthophoto to obtain a visual positioning anchor point at the first moment. The state compensation module is used to obtain the displacement increment and attitude change from the first time to the second time based on the sensing data between the first time and the second time, and to map the visual positioning anchor point to the second time to obtain the inertial compensation positioning point at the current time. A state construction module is used to construct a global navigation state based on the inertial compensation positioning point and a preset navigation endpoint. The global navigation state is then tensor-stitched with the depth image at a second time step to obtain the current state space for reinforcement learning. The construction of the global navigation state based on the inertial compensation positioning point and the preset navigation endpoint includes: obtaining the Euclidean distance between the inertial compensation positioning point and the preset navigation endpoint, and using the Euclidean distance as the distance state component; obtaining the angle between the current nose-up orientation of the UAV and the direction vector pointing from the inertial compensation positioning point to the navigation endpoint, and using this angle as the azimuth state component; obtaining the linear velocity and angular velocity of the UAV at the second time step, and using these as dynamic state components; and combining the distance state component, the azimuth state component, and the dynamic state component to generate the global navigation state. The navigation control module is used to input the current state space into the navigation policy network to obtain navigation control commands for the UAV.

Images