Unmanned aerial vehicle navigation method and system based on computer vision

Abstract

The invention relates to the technical field of unmanned aerial vehicle navigation, and discloses an unmanned aerial vehicle navigation method and system based on computer vision, and the unmanned aerial vehicle navigation method based on computer vision comprises the following steps: collecting a current visual field image of an airborne camera of an unmanned aerial vehicle, and carrying out adaptive image enhancement; performing scene semantic understanding; semantic perception feature extraction is carried out; feature matching and outer point elimination are carried out; floor identification is carried out, and multi-sensor fusion positioning is carried out; plane segmentation, structural constraint extraction and map normalization are carried out; performing visual navigation evaluation, judging a navigation mode according to an evaluation result, and generating an exploration control instruction quantity based on the navigation mode; generating a cross-floor path sequence, optimizing a track in real time, and performing control fusion in combination with a navigation mode identifier and an exploration control vector; according to the method, the navigation problem of the unmanned aerial vehicle in complex scenes such as smoke interference, multi-story buildings and illumination dramatic change in a GPS denial environment is solved, and high-precision autonomous navigation is realized.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) navigation technology, and more specifically, to a computer vision-based UAV navigation method and system. Background Technology

[0002] Traditional UAV navigation systems primarily rely on GPS satellite positioning systems for location information, with inertial measurement units (IMUs) assisting in attitude estimation and short-term position prediction. While GPS navigation boasts high positioning accuracy and global coverage in open outdoor environments, its accuracy drops drastically or even fails completely in scenarios where GPS signals are blocked or interfered with, such as urban canyons, indoor environments, and underground spaces. Although inertial navigation does not depend on external signals, it suffers from cumulative error, resulting in an exponential decline in positioning accuracy over extended periods, failing to meet the demands of long-term autonomous navigation. As UAV application scenarios continue to expand, autonomous navigation in GPS-denied environments has become a key bottleneck restricting the development of UAV technology.

[0003] Existing visual navigation technologies, including visual odometry and visual SLAM, have been applied in some scenarios. However, these methods suffer from severe performance degradation in feature extraction and matching under extreme conditions such as smoke obstruction, drastic lighting changes, and low-texture environments, making navigation systems prone to failure. In complex spatial structures such as multi-story buildings, existing methods lack effective floor identification and global consistency constraint mechanisms, leading to positioning jumps and cumulative drift. Furthermore, the information provided by a single visual sensor is limited, and effective redundancy and degradation strategies are lacking in visual degradation scenarios.

[0004] To address the aforementioned technical problems, this invention proposes a visual navigation method that integrates multispectral perception, semantic understanding, multi-sensor fusion, structural constraints, and adaptive strategies. Summary of the Invention

[0005] This invention provides a computer vision-based UAV navigation method and system, which solves the technical problems of low navigation accuracy, insufficient robustness, and lack of degradation response mechanisms in related technologies under complex environments.

[0006] This invention provides a computer vision-based unmanned aerial vehicle (UAV) navigation method, comprising:

[0007] The image of the current field of view of the UAV's onboard camera is acquired, and adaptive image enhancement is performed to obtain a multi-scale enhanced image sequence;

[0008] Scene semantic understanding is performed based on multi-scale enhanced image sequences to obtain a semantic label map; semantic perception feature extraction is performed to obtain a feature descriptor subset;

[0009] Feature matching and outlier removal are performed based on the feature descriptor subset to obtain the final set of matching point pairs; the essential matrix is ​​calculated based on the final set of matching point pairs using the RANSAC algorithm.

[0010] Singular value decomposition is performed on the essential matrix; floor identification is performed based on the semantic label map; multi-sensor fusion localization is performed by combining the final matching point set to obtain the fusion optimized pose and root mean square value of reprojection error.

[0011] Based on the fusion-optimized pose and the final matching point pair set, triangulation is performed. Combined with the semantic label map, plane segmentation, structural constraint extraction and map regularization are performed. A multi-layer map management strategy is adopted to obtain a complete multi-layer semantic map.

[0012] Visual navigation is evaluated based on the root mean square value of reprojection error. The navigation mode is determined based on the evaluation results. Exploration control commands are generated based on the navigation mode to obtain the exploration control vector.

[0013] Based on the fusion optimization of pose and the complete multi-layer semantic map, a cross-floor path sequence is generated and the trajectory is optimized in real time. The control is fused by combining navigation mode identifiers and exploration control vectors to obtain the final control command.

[0014] In a preferred embodiment, the current field-of-view image acquired from the UAV-borne camera includes the visible light RGB image and the infrared thermal image of the current field of view from the UAV-borne visible light binocular camera and infrared thermal imaging camera.

[0015] Based on visible light RGB images and infrared thermal images, synchronous acquisition is performed at a fixed frame rate, and the original visible light image sequence and the original infrared image sequence are obtained by sorting by timestamp.

[0016] Adaptive image enhancement is performed based on the original visible light image sequence and the original infrared image sequence. The adaptive image enhancement includes smoke concentration estimation, multimodal fusion, concentration adaptive defogging, contrast enhancement, and multiscale decomposition to obtain the multiscale enhanced image sequence.

[0017] In a preferred embodiment, the smoke concentration estimation includes:

[0018] Based on the original visible light image, the dark channel prior analysis method is used to perform local window scanning on the original visible light image, calculate the minimum pixel value of the three color channels of RGB in each local window, and statistically analyze the distribution characteristics of the minimum pixel value in the whole image. By analyzing the offset of the dark channel statistical histogram, the initial value of smoke concentration in the visible light field of view is estimated, and the visible light smoke concentration parameter is obtained.

[0019] Based on the original infrared image sequence, the temperature gradient analysis method is used to map the pixel gray values ​​of the original infrared image to temperature values, calculate the temperature gradient field of the original infrared image, and determine the degree of absorption and attenuation of infrared radiation by the smoke by analyzing the spatial distribution and attenuation characteristics of the temperature gradient, thereby obtaining the infrared smoke concentration parameter.

[0020] Based on visible light smoke concentration parameters and infrared smoke concentration parameters, an exposure adaptive weighted fusion algorithm is used to calculate the average brightness value of the original visible light image. The average brightness value is compared with a preset overexposure threshold. The fusion weight of the visible light mode is calculated by the Sigmoid function. The infrared mode weight is subtracted from the visible light weight. The smoke concentration parameters of the two modes are fused using a weighted average method to obtain the fused smoke concentration estimate.

[0021] In a preferred embodiment, the scene semantic understanding based on multi-scale enhanced image sequences employs a semantic segmentation network, and the pre-training process of the semantic segmentation loss function is as follows:

[0022] The total number of pixels in the original enhanced image in the multi-scale enhanced image sequence is recorded as the total number of pixels.

[0023] Determine the total number of categories in the semantic segmentation task;

[0024] For each pixel location, obtain the actual semantic category label;

[0025] For each pixel location, obtain the probability value predicted by the semantic segmentation network that this pixel belongs to each category;

[0026] For each category of each pixel, calculate the cross-entropy between the true label and the predicted probability, which is to multiply the logarithm of the predicted probability by the true label;

[0027] The cross-entropy values ​​of all categories for all pixels are summed, and a negative sign is added before the summation to obtain the negative log-likelihood loss. The negative log-likelihood loss is then divided by the total number of pixels to obtain the average semantic segmentation loss value.

[0028] In a preferred embodiment, the semantic-aware feature extraction includes semantic reliability assessment, multi-scale feature detection, semantic weighted filtering, and hybrid feature descriptor construction. The multi-scale feature detection involves performing multi-scale feature detection on each layer of the multi-scale enhanced image sequence, specifically including:

[0029] An improved FAST corner detection algorithm is used to calculate the gradient magnitude map of each image layer, and the gradient mean and standard deviation of the neighborhood of each image pixel are statistically analyzed. The corner detection threshold is adaptively set according to the local gradient statistical characteristics. The corner detection threshold is set by adding the gradient mean to the adjustment coefficient and multiplying the gradient standard deviation. The adjustment coefficient is dynamically set according to the smoke concentration.

[0030] By comparing the grayscale differences of each candidate point in each layer of the image, the initial set of corner points for each layer of the image is obtained.

[0031] In a preferred embodiment, the semantic reliability assessment, which involves removing low-weight region corner points and retaining high-reliability semantic region feature points based on the semantic reliability weight map, is specifically implemented as follows:

[0032] Based on the semantic label graph, a semantic category reliability assessment method is adopted, and different weights are assigned according to the stability and reliability of different semantic categories in visual navigation;

[0033] A semantic reliability weight map is obtained by assigning corresponding weights to each pixel position in the semantic label map according to the category.

[0034] Based on the multi-scale fusion of corner point set and semantic reliability weight map, a semantic weighted filtering strategy is adopted. The weight value of each corner point position in the weight map is read, a weight threshold is set and corner points with weights below the threshold are removed, and corner points in high-reliability semantic regions are retained first, resulting in a semantically filtered feature point set.

[0035] In a preferred embodiment, the floor identification step includes:

[0036] Based on semantic label graphs, a deep learning object detection method is used to identify floor signage information. A pre-trained lightweight object detection network is used to detect key floor targets on the semantic label graph, including floor signs, stair signs, and elevator signs.

[0037] Image cropping is performed on the key target areas of the detected floors. Optical character recognition (OCR) technology is used to identify the floor numbers on the signs. The numerical information is extracted as the visual floor recognition result to obtain the visual floor estimate.

[0038] Based on the current barometric altitude value measured by the barometric altimeter on the drone, and combined with the historical barometric altitude sequence, a sliding window height change detection method is used to determine the floor change. The difference between the current altitude and the initial altitude is calculated, and then divided by the standard floor height and rounded to obtain the floor change amount.

[0039] The barometric floor estimate is obtained by adding the floor change amount to the initial floor level.

[0040] Based on the visual floor estimate, the air pressure floor estimate, and the floor status at the previous moment, the posterior probability distribution of the current floor is calculated using the Bayesian fusion method. A likelihood model is established to evaluate the probability of observing the current measurement under a certain floor condition. Combined with the prior probability of the floor at the previous moment, the posterior probability of each floor is calculated according to the Bayesian formula. The floor with the highest posterior probability is selected as the fused floor judgment result to obtain the current floor identifier.

[0041] In a preferred embodiment, the structural constraint extraction is specifically implemented as follows:

[0042] Based on the architectural scene, the planar set is used to extract the geometric constraints of the building structure using a planar relationship analysis algorithm, and to calculate the angle between the normal vectors of any two planes and the distance between their center points.

[0043] The angle between the normal vectors of two planes determines the type of plane pair. The types of plane pairs include parallel plane pairs and perpendicular plane pairs.

[0044] The perpendicular relationship between the wall plane and the ground plane and the right angle relationship between adjacent walls are detected, and these geometric constraints are expressed as constraint equations to obtain the set of structural constraints.

[0045] In a preferred embodiment, the reliability of visual navigation is evaluated using a fuzzy inference system based on the number of feature points in the current frame, the matching inlier rate, the root mean square value of the reprojection error, and the smoke concentration parameter, and each index is fuzzified.

[0046] The number of feature points is divided into three fuzzy sets: small, medium, and large; the matching in-point rate is divided into three fuzzy sets: low, medium, and high; the reprojection error is divided into three fuzzy sets: small, medium, and large; and the smoke concentration is divided into three fuzzy sets: low, medium, and high.

[0047] Define fuzzy inference rules, calculate the fuzzy distribution of reliability scores based on the fuzzy rules and the fuzzy membership degrees of each index, and obtain the visual reliability score through defuzzification.

[0048] In a preferred embodiment, a computer vision-based unmanned aerial vehicle (UAV) navigation system is used to execute the aforementioned computer vision-based UAV navigation method, comprising:

[0049] The image processing module is used to acquire the current field-of-view image of the UAV's onboard camera, perform adaptive image enhancement, and obtain a multi-scale enhanced image sequence;

[0050] The semantic-aware feature extraction module performs scene semantic understanding based on multi-scale enhanced image sequences, obtains a semantic label map, and performs semantic-aware feature extraction to obtain a feature descriptor subset.

[0051] The feature matching module performs feature matching and outlier removal based on the feature descriptor subset to obtain the final set of matched point pairs; the essence matrix is ​​calculated based on the final set of matched point pairs using the RANSAC algorithm.

[0052] The multimodal fusion localization module performs singular value decomposition on the essential matrix; it identifies floors based on the semantic label map, and performs multi-sensor fusion localization by combining the final matching point set to obtain the fusion optimized pose and root mean square value of reprojection error;

[0053] The map building module performs triangulation based on the fusion-optimized pose and the final matching point pair set, combines the semantic label map for plane segmentation, structural constraint extraction and map regularization, and adopts a multi-layer map management strategy to obtain a complete multi-layer semantic map.

[0054] The visual degradation detection module performs visual navigation evaluation based on the root mean square value of reprojection error, determines the navigation mode based on the evaluation results, generates exploration control commands based on the navigation mode, and obtains the exploration control vector.

[0055] The intelligent path planning and control module generates cross-floor path sequences based on fused and optimized pose and complete multi-layer semantic maps, and optimizes the trajectory in real time. It combines navigation mode identifiers and exploration control vectors to perform control fusion and obtain the final control command.

[0056] The beneficial effects of this invention are as follows: By fusing image information from both visible light and infrared modalities for smoke concentration estimation and image enhancement, the problem of perception failure caused by reduced contrast and overexposure of firelight in dense smoke environments using only visible light vision is effectively solved; when the visible light image is overexposed due to firelight, the system automatically increases the fusion weight of the infrared image, utilizing the strong smoke penetration capability of infrared to maintain environmental perception; simultaneously, a lightweight semantic segmentation network is introduced to perform semantic understanding of the scene, identifying static and reliable structures such as walls and the ground, prioritizing the retention of feature points in highly reliable semantic regions during the feature extraction stage, and eliminating features from unreliable regions such as the sky and dynamic objects, thereby improving feature quality from the source; thus improving the system's perception robustness and positioning reliability in harsh environments.

[0057] This invention addresses the issues of floor confusion and positioning drift in multi-story building navigation by proposing a multimodal floor identification method that integrates visual sign recognition, barometric altitude detection, and Bayesian probabilistic inference. By detecting floor signs and combining them with barometric altitude change trends, the method achieves accurate floor identification, thus improving the accuracy of floor identification.

[0058] During the map construction process, the inherent structural regularity of buildings is fully utilized to automatically extract geometric priors such as the vertical constraints between walls and the ground and the right angle constraints between adjacent walls. These constraints are then incorporated into the optimization framework to regularize the map, effectively suppressing the scale drift and cumulative errors common in pure visual methods.

[0059] By employing a multi-layer map management strategy, point cloud and planar information for each floor are maintained separately, supporting continuous positioning across floors and global consistency optimization. Attached Figure Description

[0060] Figure 1 This is a flowchart of a computer vision-based drone navigation method according to the present invention;

[0061] Figure 2 This is a block diagram of a computer vision-based unmanned aerial vehicle (UAV) navigation system according to the present invention. Detailed Implementation

[0062] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0063] At least one embodiment of the present invention discloses a computer vision-based drone navigation method, such as... Figure 1 As shown, it includes:

[0064] Step 1: Acquire the current field-of-view image from the UAV's onboard camera, perform adaptive image enhancement, and obtain a multi-scale enhanced image sequence;

[0065] Input: Visible light RGB image and infrared thermal image of the current field of view from the visible light binocular camera and infrared thermal imaging camera on the UAV;

[0066] Based on visible light RGB images and infrared thermal images, synchronous acquisition at a fixed frame rate was performed, and the original visible light image sequence and the original infrared image sequence were obtained by sorting them by timestamp. Adaptive image enhancement was then performed on the original visible light image sequence and the original infrared image sequence. Adaptive image enhancement includes smoke concentration estimation, multimodal fusion, concentration-adaptive dehazing, contrast enhancement, and multi-scale decomposition, resulting in a multi-scale enhanced image sequence, specifically including:

[0067] Based on the original visible light image, a dark channel prior analysis method is used to perform local window scanning on the original visible light image. The minimum pixel value of the three RGB color channels within each local window is calculated, and the distribution characteristics of the minimum pixel value across the entire image are statistically analyzed. By analyzing the offset of the dark channel statistical histogram, the initial value of the smoke concentration in the visible light field of view is estimated, thus obtaining the visible light smoke concentration parameter. Specifically, for each pixel position in the original visible light image, the minimum value of the three RGB channels is extracted within a 15×15 window centered on that pixel as the dark channel value at that position. The normalized mean of the dark channel values ​​across the entire image is calculated, and the visible light smoke concentration parameter is obtained by subtracting this mean from 1. The closer the value is to 1, the denser the smoke.

[0068] Based on the original infrared image sequence, a temperature gradient analysis method is used to map the pixel grayscale values ​​of the original infrared image to temperature values, calculate the temperature gradient field of the original infrared image, and determine the degree of absorption and attenuation of infrared radiation by smoke by analyzing the spatial distribution and attenuation characteristics of the temperature gradient, thus obtaining the infrared smoke concentration parameter. Specifically, Sobel gradient calculation is performed on the temperature image to obtain a temperature gradient amplitude map, high-temperature regions are identified as fire source locations, the temperature gradient attenuation rate from the fire source to each location in the image is calculated, and an exponential attenuation model is used to fit the attenuation coefficient, which reflects the smoke concentration, thus obtaining an estimated smoke concentration value in the infrared mode.

[0069] Based on visible light and infrared smoke concentration parameters, an exposure-adaptive weighted fusion algorithm is used to calculate the average brightness value of the original visible light image. This average brightness value is compared with a preset overexposure threshold (using 95% of the normal scene brightness as the initial threshold). The fusion weight of the visible light mode is calculated using the Sigmoid function, while the infrared mode weight is 1 minus the visible light weight. A weighted average method is used to fuse the smoke concentration parameters of the two modes to obtain a fused smoke concentration estimate. When the average brightness of the original visible light image is below the overexposure threshold, the visible light weight is relatively large, around 0.7, and the infrared weight is around 0.3. When the brightness exceeds the threshold, indicating overexposure, the visible light weight rapidly decreases to below 0.3, and the infrared weight correspondingly increases to above 0.7, ensuring that when visible light overexposure is caused by fire, smoke estimation primarily relies on infrared information.

[0070] Before weighted fusion, the visible light smoke concentration parameters and infrared smoke concentration parameters need to be normalized preprocessed, mapping the two parameters to the interval [0, 1] respectively. This ensures that the concentration parameters of different modes have the same numerical range and dimensions, avoiding fusion deviations caused by differences in numerical scales. The normalization process is as follows: obtain the historical minimum and maximum values ​​of the concentration parameter for this mode; subtract the minimum value from the current concentration value, and then divide by the difference between the maximum and minimum values ​​to obtain the normalized concentration parameter.

[0071] Based on the fusion of smoke concentration estimates and the original visible light image, a concentration-adaptive atmospheric scattering model dehazing algorithm is adopted. The model parameters are dynamically adjusted according to the smoke concentration: the global atmospheric light value is estimated, the original image positions corresponding to the top 1% of pixels with the largest dark channel value are selected, the pixel value with the largest sum of RGB channels is selected as the atmospheric light in these positions, the transmittance map is calculated based on the dark channel value and smoke concentration, the transmittance map is smoothed and optimized using guided filtering, and the scene radiance value after dehazing is calculated according to the atmospheric scattering model inversion formula to obtain the dehazed image.

[0072] The steps for calculating transmittance are as follows: Obtain the basic parameters of the pixel location (including the horizontal and vertical coordinates of the pixel location and the corresponding normalized dark channel value); set a weight coefficient for preserving the fog effect, typically set to 0.95, to avoid image distortion caused by completely removing fog; obtain an estimated value of the fused smoke concentration in the current scene; multiply the weight coefficient for preserving the fog effect by the estimated smoke concentration to obtain a concentration adjustment factor; multiply the concentration adjustment factor by the normalized dark channel value of the pixel location to obtain the fog attenuation amount at that pixel location; subtract the fog attenuation amount from 1 to obtain the transmittance value of the pixel location. The closer the transmittance value is to 1, the thinner the fog at that location; the closer the transmittance value is to 0, the denser the fog at that location.

[0073] The steps for restoring the scene radiance after dehazing are as follows: 1. Obtain the original pixel value at a specified pixel location in the original visible light image; 2. Obtain the global atmospheric light value; 3. Calculate the difference between the original pixel value and the global atmospheric light value to obtain the signal intensity after removing the influence of atmospheric light; 4. Obtain the transmittance value corresponding to the specified pixel location in the original visible light image, while setting a lower limit threshold of 0.1 to avoid division by zero; 5. Compare the transmittance value with the lower limit threshold and select the larger value as the effective transmittance; 6. Divide the signal intensity after removing the influence of atmospheric light by the effective transmittance to obtain the scene radiance after dehazing; 7. Add the global atmospheric light value to the scene radiance after dehazing to obtain the final dehazed image pixel value, which restores the true brightness information of the original scene under fog-free conditions.

[0074] Based on the dehazed image, a contrast-limited adaptive histogram equalization algorithm is used to divide the image into 8×8 sub-blocks. Histograms are calculated for each sub-block and equalization transformation is performed, while limiting the maximum peak value of the histogram to avoid over-enhancement. Bilinear interpolation is used to achieve a smooth transition between sub-blocks, resulting in a contrast-enhanced image. Based on the contrast-enhanced image, a Gaussian pyramid multi-scale decomposition method is used to construct a 4-layer image pyramid. The 0th layer is the original enhanced image, and each subsequent layer is generated through 5×5 Gaussian kernel convolution and 2x downsampling, forming image representations at different resolutions, resulting in a multi-scale enhanced image sequence. Simultaneously, a fused smoke concentration estimate is output for subsequent steps.

[0075] This step outputs a multi-scale enhanced image sequence and a fused smoke concentration estimate.

[0076] Furthermore, a deep learning dehazing network can be used to replace the traditional atmospheric scattering model. Based on the fusion of smoke concentration estimates and the original visible light image, an end-to-end convolutional neural network dehazing model is adopted. The original image and smoke concentration value are used as network inputs. Image features are extracted through a multi-layer convolutional encoder, and a clear image is reconstructed through a decoder. During network training, pairs of foggy and clear images are used to construct supervision signals to obtain the deep learning dehazed image. The purpose of this method is to achieve better dehazing results under complex lighting and non-uniform smoke distribution. The network adopts a U-Net architecture. The encoder contains 5 convolutional blocks that progressively reduce the resolution to extract features, and the decoder contains 5 upsampling blocks that progressively restore the resolution. Multi-scale features are passed between the encoder and decoder through skip connections. The network output is a dehazed image with the same resolution as the input. The training loss function combines pixel-level L1 loss and perceptual loss to preserve image details.

[0077] Step 2: Perform scene semantic understanding based on multi-scale enhanced image sequences to obtain a semantic label map, and extract semantic-aware features to obtain a feature descriptor subset;

[0078] Input: Multi-scale enhanced image sequence;

[0079] Based on the multi-scale enhanced image sequence output from step 1, a semantic segmentation network is used for scene semantic understanding, outputting a semantic label map (a two-dimensional image with the same resolution as the input, where each pixel stores its semantic category index or the category with the highest probability). To adapt to different computing power and accuracy requirements, the lightweight solution uses MobileNetV3 as the backbone combined with a feature pyramid network, gradually restoring spatial resolution and performing pixel-by-pixel classification through upsampling and convolution; the high-precision solution uses the DeepLabV3+ network architecture, outputting the semantic category probability distribution of each pixel and obtaining the final semantic label through argmax operation.

[0080] The pre-training process for the semantic segmentation loss function is as follows:

[0081] The total number of pixels in the image is recorded as the total number of pixels. The total number of categories in the semantic segmentation task is determined, including all categories that need to be identified, such as walls, ground, ceiling, doors, windows, stairs, and obstacles. For each pixel location, the true semantic category label is obtained. This label is in one-hot encoded form, meaning the label value is 1 when the pixel belongs to a certain category and 0 when it does not. For each pixel location, the probability value predicted by the network for each category is obtained, and the sum of all category probabilities is 1. For each category of each pixel, the cross-entropy between the true label and the predicted probability is calculated, which is the logarithm of the predicted probability multiplied by the true label. The cross-entropy values ​​of all categories for all pixels are summed. A negative sign is added before the summation to obtain the negative log-likelihood loss. The negative log-likelihood loss is divided by the total number of pixels to obtain the average semantic segmentation loss value.

[0082] Semantic-aware feature extraction is performed, including semantic reliability assessment, multi-scale feature detection, semantic weighted filtering, and hybrid feature descriptor construction, resulting in a set of feature descriptors.

[0083] Based on the semantic label map, a semantic category reliability assessment method is adopted. Differentiated weights are assigned according to the stability and reliability of different semantic categories in visual navigation. Each pixel position in the semantic label map is assigned a corresponding weight according to its category to obtain a semantic reliability weight map. For example, static structure categories such as walls, ground, and ceiling are assigned a high weight value of 1.0, door and window categories are assigned a medium weight value of 0.6, and sky, dynamic objects, etc. are assigned a low weight value of 0.2. Before generating the semantic reliability weight map, the weight values ​​of different semantic categories need to be standardized and preprocessed. The mean and standard deviation of the weights of all categories are calculated. The original weight values ​​of each category are standardized by Z-score. The weight value is subtracted from the mean and divided by the standard deviation. The standardized weights are mapped to the interval [0, 1] by the Sigmoid function.

[0084] Based on the multi-scale enhanced image sequence output from step 1, feature extraction is performed on each layer of the pyramid image. An improved FAST corner detection algorithm is used to calculate the gradient magnitude map of the image, and the mean and standard deviation of the gradient in the neighborhood of each pixel are statistically analyzed. The corner detection threshold is adaptively set based on the local gradient statistical characteristics. The corner detection threshold is set by adding an adjustment coefficient to the mean gradient and multiplying it by the standard deviation of the gradient. The adjustment coefficient is dynamically set according to the smoke concentration; the higher the smoke concentration, the smaller the adjustment coefficient, thus lowering the detection threshold and extracting more features under low contrast. For each candidate point in the image, the grayscale difference between that point and 16 points on its circumference is compared. When the grayscale difference of more than 9 consecutive points exceeds the threshold, it is determined to be a corner point, resulting in the initial corner set for each image layer.

[0085] Based on the initial corner point set of each layer of the image, a non-maximum suppression method is used to filter corner points. The response values ​​of each corner point are compared in the 5×5 neighborhood. Only the corner points with the largest local response value are retained, and redundant corner points in dense areas are removed to obtain the corner point set after filtering for each layer. Then, the corner points of each layer are mapped back to the original image resolution and deduplicated to obtain a multi-scale fused corner point set.

[0086] Based on a multi-scale fusion of corner point sets and a semantic reliability weight map, a semantically weighted filtering strategy is adopted. The weight value of each corner point in the weight map is read, and a weight threshold of 0.5 is set. Corner points with weights below the threshold are removed, prioritizing the retention of corner points located in highly reliable semantic regions such as walls and the ground, resulting in a semantically filtered feature point set. This filtering process reduces the proportion of feature points located in unreliable regions such as the sky and dynamic objects, improving the robustness of feature matching and localization.

[0087] Based on the semantically filtered feature point set, a feature descriptor construction method fusing local binary pattern (LBP) and gradient orientation histogram (HOG) is adopted. For each feature point, a 32×32 pixel neighborhood image patch is extracted, and LBP features are calculated. The neighborhood is divided into 16 sub-regions. Within each sub-region, a rotation-invariant uniform LBP operator is used to extract texture patterns, and the histograms of each pattern are calculated to obtain a 59-dimensional LBP feature vector. Gradient orientation histogram features are calculated, and the gradient of the neighborhood is calculated and quantized into 8 directions. The gradient orientation histogram is calculated on a 4×4 spatial grid to obtain a 128-dimensional HOG feature vector. The LBP and HOG features are concatenated to obtain a 187-dimensional hybrid feature descriptor, which has strong invariance to illumination changes and rotation. The descriptors of all feature points are summarized to obtain the feature descriptor set of the current frame, and the feature point set is also output for subsequent matching.

[0088] This step outputs a semantic label graph and a subset of feature descriptors.

[0089] Furthermore, deep learning feature descriptors can be used to replace traditional hand-designed descriptors. Based on the semantically filtered feature point set, a lightweight convolutional neural network descriptor extraction model is adopted. A 32×32 neighborhood image patch is extracted for each feature point as network input, and encoded into a 128-dimensional feature vector through multiple convolutional and pooling operations. The network is trained using a triplet loss function to ensure that the descriptors of the same feature point have small distances while the descriptors of different feature points have large distances, resulting in a deep learning feature descriptor set. The aim of this method is to improve the discriminative ability and matching accuracy of feature descriptors under extreme smoke and lighting conditions. The network structure contains 4 convolutional layers and 2 fully connected layers, with approximately 500KB of parameters. The inference speed reaches 5000 feature points per second on the Jetson platform.

[0090] Step 3: Perform feature matching and outlier removal based on the feature descriptor subset to obtain the final set of matching point pairs; calculate the essence matrix based on the final set of matching point pairs using the RANSAC algorithm.

[0091] Input: Feature descriptor subsets of the current frame and the previous frame, and feature point coordinates.

[0092] The system maintains an inter-frame data caching mechanism. After step 2 of each round of processing is completed, the set of feature descriptors and the set of feature points output by the current frame are stored in the cache area, including the pixel coordinates of the feature points, the corresponding 187-dimensional hybrid feature descriptors, and timestamp information. In the next round of processing, step 3 reads the feature information of the previous frame from the cache as a reference frame for matching, so as to realize the continuous tracking and matching of inter-frame features and ensure the temporal continuity of visual odometry.

[0093] Based on the feature descriptor set of the current frame and the feature descriptor set of the previous frame, the FLANN fast nearest neighbor search algorithm is used to establish feature matching relationships. For each descriptor of the current frame, the nearest and second nearest neighbors with the smallest Euclidean distance are searched in the descriptor set of the previous frame. The search process is accelerated by constructing a KD tree index. The feature points of the current frame are paired with their corresponding nearest neighbor feature points of the previous frame to obtain an initial set of matching point pairs.

[0094] Before performing feature descriptor matching, the descriptors need to undergo L2 normalization preprocessing, normalizing the magnitude of each descriptor vector to 1. The normalization process involves: calculating the sum of squares of all elements in the descriptor vector, taking the square root of the sum to obtain the L2 norm of the vector, and dividing each element of the descriptor vector by this L2 norm to obtain the normalized unit descriptor vector. This ensures that descriptors of different feature points have the same numerical scale, improving the accuracy of Euclidean distance calculation and the stability of matching, and avoiding matching bias caused by differences in descriptor magnitudes.

[0095] Based on the initial set of matching pairs, the Lowe's ratio test is used for the first round of matching filtering. The ratio of the nearest neighbor distance to the second nearest neighbor distance of each matching pair is calculated, and a ratio threshold of 0.75 is set. Matching pairs with a ratio greater than the ratio threshold are removed. The nearest and second nearest neighbor distances of the removed matching pairs are too close, resulting in high matching ambiguity and easy mismatch. Matching pairs with a ratio less than the ratio threshold and strong discriminative power are retained, resulting in the set of matching pairs after the ratio test.

[0096] Based on the set of matching point pairs after the ratio test, a second round of screening is performed using motion consistency constraints. The motion vector of each matching point pair is calculated, and the motion vector is defined as the coordinates of the feature point in the current frame minus the coordinates of the feature point in the previous frame. The median vector of all motion vectors is calculated, and the deviation between the motion vector and the median vector of each matching pair is calculated. Abnormal matches with a deviation greater than the deviation threshold are removed (set as 2% of the image diagonal length, which is about 16 pixels for a 640×480 resolution image and is adaptively adjusted according to the image resolution). These abnormal matches are usually caused by dynamic objects or mismatches. Matching pairs with consistent motion are retained, resulting in the set of matching point pairs after motion consistency screening.

[0097] Based on the set of matching point pairs after motion consistency screening, a third round of refinement is performed using the RANSAC algorithm based on epipolar geometric constraints. Eight pairs of matching points are randomly sampled from the matching set, and the fundamental matrix is ​​calculated using the eight-point method. The algebraic error of all matching point pairs is calculated according to the epipolar constraint equation. The number of inliers with algebraic errors less than the error threshold is counted. The sampling and calculation process is repeated 1000 times. The model with the most inliers is selected as the optimal fundamental matrix, and the set of inliers corresponding to this model is used as the refined matching point pairs to obtain the final set of matching point pairs. The ratio of the number of inliers to the total number of matching points is used to obtain the matching inlier rate. The initial value of the error threshold is set to 1.0 pixel, and it is dynamically adjusted according to the matching inlier rate. When the inlier rate is higher than 60%, it is tightened to 0.8 pixels, and when it is lower than 40%, it is relaxed to 1.5 pixels.

[0098] The verification steps for the epipolar constraint are as follows: Obtain the image coordinates of a feature point in the current frame and convert them to homogeneous coordinates (by adding the value 1 to the original two-dimensional coordinates to form a three-dimensional homogeneous coordinate vector); obtain the image coordinates of the corresponding feature point in the previous frame and convert them to homogeneous coordinates as well; obtain the fundamental matrix calculated using the eight-point method (a 3×3 matrix that encodes the geometric relationship between the two camera views); transpose the homogeneous coordinate vector of the current frame feature point to obtain a row vector; multiply the transposed current frame feature point coordinates with the fundamental matrix; perform matrix multiplication again with the homogeneous coordinate vector of the corresponding feature point in the previous frame; check if the final calculation result is close to zero. If the result is close to zero, it indicates that the matching point pair satisfies the epipolar constraint and is a correct match; otherwise, it is an incorrect match and needs to be discarded.

[0099] The methods for setting the ratio threshold, deviation threshold, and error threshold are as follows:

[0100] Ratio threshold setting: Lowe's recommended empirical value of 0.75 is used as the benchmark, and fine-tuned according to the complexity of the scene. In scenes with rich textures, it is set to 0.7 to improve matching accuracy, and in scenes with sparse textures, it is set to 0.8 to retain more matching pairs, ensuring robustness of matching under different environmental conditions.

[0101] Deviation threshold setting: determined based on image resolution and camera motion characteristics, the calculation formula is 2% of the image diagonal length, which is about 16 pixels for a 640×480 resolution image. This threshold can eliminate obvious motion anomalies while tolerating normal camera motion changes and feature localization errors.

[0102] Error threshold setting: An adaptive strategy is adopted, with an initial value of 1.0 pixels. The threshold is dynamically adjusted according to the current matching quality. When the matching inlier rate is higher than 60%, the threshold is tightened to 0.8 pixels to improve accuracy. When the inlier rate is lower than 40%, the threshold is widened to 1.5 pixels to maintain a sufficient number of inliers and ensure the geometric verification effect under different matching qualities.

[0103] Based on the optimal fundamental matrix and camera intrinsic parameter matrix obtained through RANSAC selection, the essential matrix calculation method is adopted. The essential matrix is ​​calculated through matrix multiplication. The calculation steps are as follows: Obtain the camera intrinsic parameter matrix, which contains intrinsic parameters such as the camera's focal length and principal point coordinates, and is a 3×3 matrix; Transpose the camera intrinsic parameter matrix to obtain the transposed matrix; Obtain the optimal fundamental matrix obtained through the RANSAC algorithm, which encodes the geometric constraint relationship between the two camera viewpoints; Perform matrix multiplication between the transposed camera intrinsic parameter matrix and the fundamental matrix to obtain an intermediate result matrix; Perform matrix multiplication between the intermediate result matrix and the original camera intrinsic parameter matrix to obtain the final essential matrix. This matrix eliminates the influence of the camera intrinsic parameters and purely reflects the relative rotation and translation relationship between the two cameras, providing a geometric constraint basis for subsequent pose recovery.

[0104] This step outputs the matching inlier rate, the final set of matching point pairs, and the essential matrix. Each pair of matching points in the final set contains the pixel coordinates of the current frame and the previous frame, and the essential matrix encodes the camera's relative rotation and translation information.

[0105] Furthermore, deep learning matching networks can be used to replace traditional descriptor matching methods. Based on the feature point neighborhood image patches of the current frame and the previous frame, a Siamese convolutional neural network is used to calculate the similarity between feature point pairs. The network contains two branches with shared weights that process image patches from the two frames respectively. After extracting features through convolutional layers, the cosine similarity of the feature vectors is calculated, and the matching confidence score between 0 and 1 is output. Feature point pairs with confidence scores greater than a threshold are selected as initial matches, and then the same ratio test, motion consistency constraint, and RANSAC algorithm are used for progressive filtering to obtain a refined set of matching point pairs assisted by deep learning. The purpose of this method is to leverage the powerful representational capabilities of deep learning to improve matching performance under difficult conditions. The network adopts a lightweight design with each branch containing 3 convolutional layers. The training data includes positive and negative sample pairs. Positive samples are image patches with the same feature point in different frames, and negative samples are image patches with different feature points. The loss function uses contrastive loss to drive the network to learn discriminative feature representations.

[0106] Step 4: Perform singular value decomposition on the essential matrix; perform floor identification based on the semantic label map; combine the final matching point set to perform multi-sensor fusion localization to obtain the fusion optimized pose and root mean square value of reprojection error.

[0107] Inputs: the essential matrix and final set of matching point pairs output from step 3, IMU angular velocity and acceleration data, barometric altitude data, and semantic label map output from step 2.

[0108] Based on the essential matrix output in step 3, the relative rotation and translation of the cameras are recovered using singular value decomposition (SVD). SVD is performed on the essential matrix, decomposing it into the product of the transposes of three matrices U, Σ, and V. Four possible solutions for the rotation matrix and translation vector are constructed based on the properties of the essential matrix. Triangulation tests are used to verify which solution allows the 3D coordinates of the matching point to be in front of both cameras. The unique solution satisfying the positive depth constraint is selected as the relative pose for visual estimation, yielding the relative rotation matrix and translation vector. The rotation matrix represents the pose change of the current frame relative to the previous frame, while the translation vector represents the relative position change but with an unknown scale, requiring subsequent fusion to determine the absolute scale.

[0109] Based on the semantic label map output in step 2, a deep learning object detection method is used to identify floor signage information. A pre-trained lightweight object detection network is used to detect key objects such as floor signs, stair signs, and elevator signs on the semantic label map. The detected floor signage areas are cropped, and optical character recognition (OCR) technology is used to identify the floor numbers on the signs. The numerical information is extracted as the visual floor recognition result to obtain the visual floor estimate. When no clear floor signage is detected in the image, the visual floor estimate is marked as invalid.

[0110] Based on the current barometric altitude value measured by the barometric altimeter on the UAV and the historical barometric altitude sequence, a sliding window height change detection method is used to determine the floor change. The difference between the current height and the initial height is calculated, divided by the standard floor height of 3 meters and rounded to obtain the floor change amount. The initial floor is added to the floor change amount to obtain the barometric floor estimate.

[0111] Based on visual floor estimates, barometric floor estimates, and the floor status from the previous time step, a Bayesian fusion method is used to calculate the posterior probability distribution of the current floor. A likelihood model is established to evaluate the probability of observing the current measurement under a certain floor condition. Combined with the prior probability of the floor from the previous time step, the posterior probability of each floor is calculated according to the Bayesian formula. The floor with the highest posterior probability is selected as the fused floor judgment result, thus obtaining the current floor identifier. Specifically, when the visual floor estimate and the barometric floor estimate are consistent, the posterior probability of that floor is close to 1; when they are inconsistent, they are fused by weighting according to their respective confidence levels. When the confidence level of visual recognition is high, the visual result is trusted more; when the barometric pressure changes significantly, the barometric pressure result is trusted more.

[0112] Based on angular velocity and acceleration data measured by the UAV's IMU sensor, an IMU pre-integration method is used to calculate the relative pose change within a time window. Quaternion integration of angular velocity yields the attitude change, and double integration of acceleration yields the position change. Simultaneously, the effects of IMU noise and bias are accumulated to obtain the covariance matrix, resulting in the IMU-predicted relative pose and uncertainty. IMU pre-integration is performed between adjacent keyframes, avoiding frequent state updates and improving computational efficiency.

[0113] Based on IMU sensor data, an extended Kalman filter is used for inertial navigation calculation to establish a UAV motion state model. The state vector contains 15-dimensional state variables, including position, velocity, attitude angle, gyroscope bias, and accelerometer bias. The UAV's pose state is recursively estimated through two steps: prediction and update. The prediction step predicts the state at the next moment based on IMU measurements and the motion model, while the update step uses visual measurement information to correct the prediction results.

[0114] A nonlinear state transition function is applied to perform state propagation calculations using the state estimate from the previous time step and the IMU input at the current time step. The specific implementation process of the nonlinear state transition function is as follows: 15 state components, including position, velocity, attitude quaternions, gyroscope bias, and accelerometer bias, are extracted from the state vector; state propagation is performed using IMU measurements; position updates are achieved through velocity integration; velocity updates are achieved through acceleration integration after bias removal and considering gravity compensation; attitude updates are achieved through quaternion multiplication combined with angular velocity after bias removal; the bias term is slowly varied using a random walk model; and finally, a predicted 15-dimensional state vector is output.

[0115] Based on the relative pose estimated by vision, the relative pose predicted by IMU, the current floor marker, and the set of matching point pairs output from step 3, a nonlinear optimization method is used for multi-sensor fusion localization. A factor graph optimization framework is constructed, where nodes represent pose states at different times, edges represent sensor constraints, visual constraints are modeled through reprojection error, IMU constraints are modeled through pre-integrated residuals (containing three components: rotation, velocity, and position, with the difference between the IMU pre-integration result and the optimization variable calculated separately), and floor constraints are modeled through height error. The objective function is the weighted sum of squares of all constraint residuals. The optimal pose sequence that minimizes the objective function is solved using the Levenberg-Marquardt iterative algorithm to obtain the fused optimized pose.

[0116] The reprojection error calculation process under visual constraints includes the following steps: obtaining the coordinates of observed image feature points and their corresponding 3D spatial point coordinates; obtaining the camera intrinsic parameter matrix, rotation matrix, and translation vector; applying the projection function to project the 3D spatial points onto the image plane through the camera parameters to obtain the theoretical projection coordinates; calculating the difference between the observed coordinates and the theoretical projection coordinates to obtain the reprojection residual vector; outputting the reprojection residuals and constructing the optimization objective function; and taking the square root of the average of the sum of squares of all reprojection residuals to obtain and output the root mean square value of the reprojection error.

[0117] The specific implementation of the projection function includes the following steps: transforming the 3D spatial point P to the camera coordinate system through the camera extrinsic transformation matrix to obtain the 3D point coordinates in the camera coordinate system, including X-axis, Y-axis and Z-axis components; projecting the 3D point onto the image plane through perspective projection and calculating the normalized image coordinates; converting the normalized coordinates into pixel coordinates through the camera intrinsic parameter matrix.

[0118] The pre-integration residual calculation process for IMU constraints includes three parts: rotation, velocity, and position. The rotation residual is calculated by comparing the rotational changes obtained from IMU pre-integration with the rotational changes in the optimization variables. The velocity residual is calculated by comparing the velocity changes obtained from IMU pre-integration with the velocity changes in the optimization variables. The position residual is calculated by comparing the position changes obtained from IMU pre-integration with the position changes in the optimization variables. The three residual components are combined into a complete IMU pre-integration residual vector. The IMU constraint residual is then output for constructing the subsequent optimization objective function.

[0119] The process of calculating the floor height constraint residual includes the following steps: obtaining the vertical height component of the current pose, the current floor identifier, the standard floor height parameter (set to a fixed value of 3 meters), and the initial height offset; obtaining the floor constraint residual by multiplying the floor identifier by the standard floor height and adding the initial offset; calculating the difference between the actual height and the theoretical floor height to obtain the floor constraint residual; and outputting the floor constraint residual for subsequent optimization objective function construction.

[0120] The process of constructing the optimization objective function includes the following steps: collecting the residuals of all visual constraints, calculating the squared value of each visual constraint residual, and multiplying it by the corresponding weight coefficient; collecting the residuals of all IMU constraints, calculating the squared value of each IMU constraint residual, and multiplying it by the corresponding weight coefficient; collecting the residuals of all floor constraints, calculating the squared value of each floor constraint residual, and multiplying it by the corresponding weight coefficient; summing all weighted squared residuals to obtain the overall optimization objective function value; solving for the optimal pose parameters by minimizing this objective function value; and outputting the fused optimized pose as the final result of multi-sensor fusion localization.

[0121] This step outputs the fusion-optimized pose, the current floor indicator, and the root mean square value of the reprojection error.

[0122] Furthermore, a tightly coupled visual-inertial odometry (VIO) approach can be used instead of the loosely coupled fusion method. Based on the matched point pairs output from step 3 and the raw IMU measurement data, a sliding window optimized VIO algorithm is employed to jointly optimize the pose state, feature point depth, and IMU bias within a fixed-size time window. Visual reprojection constraints and IMU pre-integration constraints are tightly coupled within a unified optimization framework. An edge-mapping strategy is used to handle historical states exceeding the window to maintain real-time performance, resulting in tightly coupled optimized pose and map point coordinates. The aim of this method is to fully utilize the complementarity of visual and inertial information, providing short-term pose predictions from the IMU when vision degrades, and providing absolute observation corrections from vision when the IMU drifts, thereby improving the continuity and accuracy of positioning.

[0123] Step 5: Based on the fusion-optimized pose and the final set of matching point pairs, triangulation is performed. Combined with the semantic label map, plane segmentation, structural constraint extraction and map regularization are performed. A multi-layer map management strategy is adopted to obtain a complete multi-layer semantic map.

[0124] Input: fused and optimized pose, final set of matched point pairs, semantic label graph.

[0125] Based on the fused and optimized pose output from step 4 and the refined set of matching point pairs output from step 3, the 3D spatial coordinates of the feature points are calculated using triangulation. For each pair of matching points, an observation ray is established from the centers of the two cameras to the feature point. The 3D position of the feature point is obtained by solving for the nearest point of the two rays or by least-squares optimization. After triangulation of all matching points, the 3D point cloud contributed by the current frame is obtained. For feature points continuously tracked across multiple frames, a multi-view triangulation method is used to jointly optimize the poses and image coordinates of all observation frames, improving the accuracy of the 3D point coordinates and obtaining a sparse 3D point cloud.

[0126] Based on the sparse 3D point cloud and the semantic label map output in step 2, a semantically assisted plane segmentation method is used to extract the building structure planes. The 3D points are grouped according to the semantic labels. Points belonging to the same semantic category and spatially close are clustered into one class. For each class of point cloud, the RANSAC plane fitting algorithm is used. Three points are randomly sampled to determine candidate planes. The distance from all points to the candidate planes is calculated. The number of interior points whose distance is less than the plane distance threshold is counted. The plane with the most interior points is iteratively selected as the fitting plane for that class of point cloud. The normal vector of the plane and the distance to the origin are extracted as plane parameters. The planes extracted from all semantic categories are summarized to obtain the set of planes of the building scene, including the main structural planes such as walls, ground, and ceiling. Each plane is associated with its corresponding semantic category. The plane distance threshold is set according to the point cloud density and measurement accuracy, and is usually set to 2-3 times the average spacing of the point cloud, about 0.1 to 0.2 meters.

[0127] Based on the set of planes in the architectural scene, a planar relationship analysis algorithm is used to extract the geometric constraints of the building structure. The angle between the normal vectors of any two planes and the distance between their center points are calculated. When the angle between the normal vectors of two planes is close to 0 degrees or 180 degrees and the distance between their center points is within 0.3m to 8m, they are considered parallel planes. When the angle is close to 90 degrees, they are considered perpendicular planes. Specifically, the perpendicular relationship between wall planes and the ground plane, and the right angle relationship between adjacent walls are detected. These geometric constraints are expressed as constraint equations, resulting in a set of structural constraints (containing knowledge of the geometric regularities of the building). Specifically, parallel constraints require the normal vectors of the two planes to be parallel, perpendicular constraints require the dot product of the normal vectors to be zero, and the Manhattan world assumption requires the normal vectors of the principal planes to be parallel to the three axes of the world coordinate system.

[0128] Based on sparse 3D point clouds, a set of planes in a building scene, and a set of structural constraints, a constraint optimization method is used for map normalization. A map normalization optimization objective function is constructed, which includes a point-to-plane distance term and a structural constraint violation penalty term. The point-to-plane distance term measures whether the points in the point cloud are well distributed on their respective planes, while the structural constraint penalty term measures whether the plane parameters satisfy geometric constraints such as parallelism and perpendicularity. Through nonlinear least squares optimization, the point cloud coordinates and plane parameters are adjusted simultaneously to make the point cloud better fit the plane and satisfy the structural constraints between planes, resulting in a structurally normalized optimized point cloud and optimized set of planes.

[0129] The process of constructing the objective function for map normalization optimization includes the following steps: Calculate the distance from all 3D points to their respective planes, using the point-to-plane distance formula, squaring each distance and summing them to obtain the point cloud fitting term; calculate the degree of violation of all structural constraints, for parallel constraints, calculate the magnitude of the inner product of the normal vectors of the two planes, for perpendicular constraints, calculate the absolute value of the inner product of the normal vectors, for Manhattan constraints, select the direction closest to the coordinate axes, squaring the degree of violation of each constraint and summing them to obtain the constraint penalty term; multiply the constraint penalty term by a weight coefficient and add it to the point cloud fitting term to obtain the overall optimization objective function and perform a minimization operation to solve for the optimal optimized point cloud coordinates and plane parameters.

[0130] Based on optimized point cloud coordinates and current floor identifiers, a multi-layer map management strategy is adopted. Point clouds are assigned to corresponding floors according to their 3D height coordinates and floor identifiers. An independent point cloud and planar set are maintained for each floor, and a floor index is established for quick querying and updating. For point clouds located at floor boundaries, such as staircases and elevators, cross-layer links are established with adjacent floors. While constructing the multi-layer semantic map, vertical planes (walls, columns, etc.) and horizontal planes (ground, ceiling, etc.) in the optimized planar set are categorized and stored according to semantic categories, forming a local obstacle information layer for each floor. This information layer, as a component of the multi-layer semantic map, contains the location, geometry, and semantic attributes of all static obstacles within the current floor, resulting in a complete multi-layer semantic map. This map structure supports efficient localization and path planning within a single floor, as well as navigation and relocation across floors.

[0131] This step outputs a complete multi-layer semantic map, including 3D point clouds, planar parameters, semantic labels, cross-layer link information, and local obstacle information for each floor. The local obstacle information includes the position and geometry of static obstacles such as walls and pillars around the current location.

[0132] Furthermore, octree map representation can be used instead of sparse point cloud representation. Based on optimized point cloud and semantic labels, an octree voxelization method is used to divide the 3D space into voxel grids of different resolutions. For each voxel, the occupancy probability, semantic category, and planar parameters are stored. Multiple observations are fused through a probability update mechanism to dynamically adjust the voxel occupancy state, resulting in a dense octree semantic map. The aim of this method is to provide a more complete representation of the environment's geometry, supporting dense collision detection and path planning. The multi-resolution characteristics of octrees can balance map accuracy and storage efficiency. High-resolution voxels are used in obstacle-dense areas, while low-resolution voxels are used in open areas. The update weight of the voxel occupancy probability is adjusted according to the uncertainty of the observation.

[0133] Step 6: Perform visual navigation evaluation based on the root mean square value of reprojection error, determine the navigation mode based on the evaluation results, generate exploration control commands based on the navigation mode, and obtain the exploration control vector;

[0134] Inputs: Matching inlier rate, root mean square value of reprojection error, and smoke concentration parameters.

[0135] Based on the number of feature points in the current frame, the matching inlier rate obtained in step 3, the root mean square value of the reprojection error calculated in step 4, and the smoke concentration parameter output in step 1, a fuzzy inference system is used to evaluate the reliability of visual navigation. Each indicator is fuzzified: the number of feature points is divided into three fuzzy sets (few, medium, and many); the matching inlier rate is divided into three fuzzy sets (low, medium, and high); the reprojection error is divided into three fuzzy sets (small, medium, and large); and the smoke concentration is divided into three fuzzy sets (low, medium, and high). Fuzzy inference rules are defined (e.g., "if the number of feature points is small and the matching inlier rate is low, then the reliability is low"). The fuzzy distribution of the reliability score is calculated based on the fuzzy rules and the fuzzy membership degree of each indicator. The visual reliability score is obtained through defuzzification.

[0136] Before performing fuzzy inference, the input indicators need to be standardized to ensure comparability of indicators with different dimensions. The number of feature points is standardized by performing a logarithmic transformation: first, add 1 to the number of feature points, take the logarithm, subtract the logarithmic mean, and then divide by the logarithmic standard deviation. The matching inlier rate is standardized linearly by subtracting the minimum value and dividing by the difference between the maximum and minimum values. The reprojection error is standardized by performing an inverse transformation: calculate 1 divided by 1 plus the ratio of the error value to the reference error value. The smoke concentration is standardized linearly by subtracting the minimum value and dividing by the difference between the maximum and minimum values. All standardized indicator values ​​are mapped to the [0, 1] interval to ensure the consistency of the fuzzy membership function and the stability of the inference results.

[0137] Based on visual reliability scores, a threshold grading strategy is adopted to determine the navigation mode. A high threshold of 0.7 and a low threshold of 0.3 are set. When the reliability score is greater than 0.7, visual navigation is considered reliable, and a pure visual navigation mode is selected, mainly relying on visual positioning with the IMU only used as an auxiliary. When the score is between 0.3 and 0.7, it is considered moderately degraded, and the system switches to a tightly coupled visual-inertial mode, where visual and IMU are equally integrated to provide pose estimation. When the score is less than 0.3, it is considered severely degraded, and the system switches to an IMU-dominated inertial extrapolation mode, mainly relying on the IMU to perform short-term dead reckoning to maintain navigation continuity and obtain the current navigation mode identifier.

[0138] Based on the current navigation mode identifier, when in IMU-dominated mode, an active exploration strategy is employed to generate exploration control commands. This involves analyzing the spatial patterns of feature point distribution in historical image sequences to identify which flight directions and attitude angles have a high number of feature points. Control increments are then generated to adjust the UAV's yaw, pitch, and altitude, guiding the UAV to actively seek out feature-rich areas. For example, the UAV may move closer to walls, decrease altitude to increase ground texture, or rotate to find corners with prominent features, resulting in an exploration control vector. This control vector is then superimposed on the normal navigation control, enabling the UAV to actively adjust its attitude and restore vision while maintaining a general flight direction.

[0139] Based on the historical sequence of visual reliability scores, a sliding window trend detection method is used to monitor visual recovery. The linear regression slope of the scores over the past 10 time points is calculated. When the slope is greater than a positive threshold, it indicates that the score is continuously rising. When the current score exceeds the recovery threshold of 0.5, it is determined that visual navigation has recovered from the degraded state. A mode switching signal is generated to switch from IMU-dominated mode back to visual-inertial fusion mode. At the same time, the relocalization process is triggered to detect the stored map features in the current image and re-establish the global pose, ensuring a smooth transition of mode switching.

[0140] This step outputs the current navigation mode identifier and exploration control vector.

[0141] Furthermore, a deep learning-based visual quality assessment network can be used to replace the fuzzy inference system. Based on the current frame image and feature distribution map, a convolutional neural network is used to directly predict the reliability score of visual navigation. The network input includes the original image, feature point location heatmap, and optical flow map. High-level semantic features of visual quality are extracted through multiple convolutions, and a reliability score between 0 and 1 is output after passing through a fully connected layer. The network is trained on a large amount of labeled data, which includes images under various degradation scenarios and their corresponding real navigation performance, to obtain the reliability score evaluated by deep learning. The purpose of this method is to leverage the powerful fitting ability of deep learning to learn complex visual quality patterns, which can more accurately predict navigation performance compared to manually designed fuzzy rules. The network uses ResNet18 as the backbone network, and the last fully connected layer outputs a single score value. The training loss function is mean squared error, and data augmentation includes adding different levels of smoke, motion blur, and illumination changes.

[0142] Step 7: Generate cross-floor path sequences based on fused and optimized pose and complete multi-layer semantic map, and optimize trajectory in real time. Combine navigation mode identifier and exploration control vector to perform control fusion and obtain final control command.

[0143] Input: fused optimized pose, complete multi-layer semantic map, navigation pattern identifier and exploration control vector.

[0144] Based on the fusion-optimized pose output from step 4, the complete multi-layer semantic map output from step 5, the predetermined target location, and the current floor identifier, a hierarchical path planning method is adopted. At the topology layer, coarse cross-floor planning is performed. The system determines whether a floor switch is needed based on the current floor and the target floor. If a switch is required, the system finds staircases or elevators in the map as intermediate nodes, generating a cross-floor path sequence containing the floor switch points. At the geometry layer, fine planning is performed within a single floor. For each single-floor path segment in the cross-floor path sequence, the Fast Mover Method (FMM) is used to calculate the shortest path from the current location to the target point on the grid map. The FMM obtains the arrival time of each grid cell by solving the propagation wavefront equation, backtracks along the gradient descent direction to generate the shortest path, and connects the paths to obtain the complete global geometric path. This results in a global reference path from the current location to the target location, containing a series of 3D path points and corresponding floor identifiers. The path avoids obstacle elements in the map and is distributed along the center lines of passageways and rooms.

[0145] Based on a global reference path and fusion-optimized pose, a local window extraction method is employed. A 10-meter segment of the path is extracted from the current position, and the path points of this segment are extracted as the reference trajectory for local planning, resulting in a local reference path. The local window dynamically slides as the UAV moves forward, always keeping the planning area in front of the UAV, thus balancing planning scope and real-time performance.

[0146] Based on the local reference path, the fusion optimized pose and the current 3D velocity vector of the UAV, and the local obstacle information in the complete multi-layer semantic map from step 5, the Model Predictive Control (MPC) method is used for real-time trajectory optimization. A dynamic prediction model of the UAV is established, and state prediction is performed in a 5-second prediction time domain with a time step of 0.1 seconds. The state includes position, velocity, and acceleration. The control input is acceleration commands in three directions. A quadratic programming (QP) objective function is constructed, which includes trajectory tracking error term, control smoothing term, velocity penalty term, and obstacle rejection term. Solving the QP problem yields the optimal control sequence. The first control variable in the sequence is taken as the execution control at the current moment to obtain the optimal velocity and acceleration commands.

[0147] The MPC objective function construction process is as follows: the squared difference between the predicted position and the reference position is used as the tracking error term, the squared control input vector is used as the smoothing term, the squared velocity vector is used as the velocity penalty term, and the repulsion term is calculated using an exponential decay function based on the distance from the predicted position to the obstacle. The total objective function is obtained by weighted summation of the terms, and the optimal control sequence is solved by minimizing this function.

[0148] Obstacle penalty function implementation: Calculates the minimum distance from the current position to all known obstacles, and uses an exponential decay function to calculate the penalty value, ensuring the drone maintains a safe distance from obstacles. Obstacle distances are quickly obtained by querying a pre-built distance field or octree map.

[0149] MPC constraints include: velocity constraints to ensure flight speed remains within a safe range, acceleration constraints to ensure control commands are within the actuator's physical capabilities, and safe distance constraints to ensure the predicted trajectory maintains sufficient distance from obstacles. The distance function calculates Euclidean distances by traversing the map and obstacles, and uses bounding boxes to simplify complex geometries to improve computational efficiency.

[0150] Based on the optimal speed and acceleration commands, a cascaded PID controller is used to convert the high-level commands into motor control quantities. The outer loop position controller calculates the desired speed based on the position error, the middle loop speed controller calculates the desired acceleration and attitude angle based on the speed error, and the inner loop attitude controller calculates the torque command based on the attitude error. Finally, the thrust and torque commands are converted into the speed commands of the four motors through the control allocation matrix to obtain the motor control vector.

[0151] Based on the navigation mode identifier and exploration control vector output in step 6, a mode-dependent control fusion strategy is adopted. When in normal navigation mode, the motor control vector is directly used as the final output. When in exploration mode, the exploration control vector is converted into attitude angle increments and superimposed and fused with the motor control vector. The fusion weight is dynamically adjusted according to the degree of visual degradation. The more severe the degradation, the greater the exploration control weight, and the final control command after fusion is obtained.

[0152] This step outputs the final control command, which is then sent to the UAV flight control system for execution, completing one navigation cycle. This process repeats from steps 1 to 7 to achieve continuous autonomous navigation for the UAV.

[0153] Furthermore, an end-to-end control strategy based on deep reinforcement learning can be used to replace the hierarchical planning and control methods. Based on the current image, map information, target location, and current state, a deep neural network strategy directly outputs control commands. The network input includes image features, a grid representation of the local map, the target's relative position, and its current velocity and attitude. Convolutional layers process the image and map, while fully connected layers fuse multimodal information to output control quantities for four motors. The network is trained in a simulation environment using reinforcement learning, with a reward function including target proximity rewards, collision penalties, and control smoothness rewards, resulting in an end-to-end control strategy network. The aim of this method is to simplify the control process and improve real-time performance. It leverages the high-dimensional mapping capabilities of deep learning to directly learn the mapping relationship from perception to control, avoiding explicit path planning and trajectory optimization. The network adopts an Actor-Critic architecture, where the Actor network outputs control commands, and the Critic network evaluates state value. The training algorithm is PPO (Proximal Policy Optimization), and the training data comes from automatic data collection in the Gazebo simulation environment. The simulation scenario includes various building layouts, smoke conditions, and dynamic obstacles.

[0154] A computer vision-based drone navigation system, such as Figure 2 As shown, a computer vision-based UAV navigation method for performing the above-described method includes:

[0155] The image processing module is used to acquire the current field-of-view image of the UAV's onboard camera, perform adaptive image enhancement, and obtain a multi-scale enhanced image sequence;

[0156] The semantic-aware feature extraction module performs scene semantic understanding based on multi-scale enhanced image sequences, obtains a semantic label map, and performs semantic-aware feature extraction to obtain a feature descriptor subset.

[0157] The feature matching module performs feature matching and outlier removal based on the feature descriptor subset to obtain the final set of matched point pairs; the essence matrix is ​​calculated based on the final set of matched point pairs using the RANSAC algorithm.

[0158] The multimodal fusion localization module performs singular value decomposition on the essential matrix; it identifies floors based on the semantic label map, and performs multi-sensor fusion localization by combining the final matching point set to obtain the fusion optimized pose and root mean square value of reprojection error;

[0159] The map building module performs triangulation based on the fusion-optimized pose and the final matching point pair set, combines the semantic label map for plane segmentation, structural constraint extraction and map regularization, and adopts a multi-layer map management strategy to obtain a complete multi-layer semantic map.

[0160] The visual degradation detection module performs visual navigation evaluation based on the root mean square value of reprojection error, determines the navigation mode based on the evaluation results, generates exploration control commands based on the navigation mode, and obtains the exploration control vector.

[0161] The intelligent path planning and control module generates cross-floor path sequences based on fused and optimized pose and complete multi-layer semantic maps, and optimizes the trajectory in real time. It combines navigation mode identifiers and exploration control vectors to perform control fusion and obtain the final control command.

[0162] In one embodiment of the present invention, a specific example is provided:

[0163] This invention was applied to a fire rescue mission in a 20-story residential building in a city. A fire broke out on the 3rd floor, and smoke spread to the 5th floor. A drone entered from a 1st-floor window, aiming to reach the 4th floor to search for trapped individuals. The drone was equipped with a 640×480 resolution binocular camera, a 320×240 resolution infrared camera, an IMU, and a barometer. The onboard platform was a Jetson Xavier NX.

[0164] Upon entering the building, the drone collected images in the first-floor corridor. Step 1 detected a smoke concentration of 0.3, dehazed and enhanced the image, generating a 4-layer pyramid. Step 3 semantically segmented and identified the corridor walls and floor, extracting 537 feature points. After semantic filtering, 421 highly reliable feature points were retained. Step 4 matched the image with the previous frame, obtaining 312 initial matches. After ratio testing, 276 pairs were retained, and after RANSAC, 201 pairs of interior points were retained. The essential matrix was calculated. Step 5 visually recognized the "First Floor" sign, with an air pressure altitude of 5.2 meters. The system fused and determined that the user was currently on the first floor. Visual inertial optimization was then performed to obtain the positioning information. The pose is (x=2.3m, y=0.8m, z=0.5m); Step 5: Construct a 3D point cloud of the corridor with a total of 1523 points, extract 2 wall planes and 1 ground plane, and detect the vertical constraint between the wall and the ground; Step 6: Calculate the number of feature points (421), inlier rate (64.4%), reprojection error (0.8 pixels), smoke concentration (0.3), and obtain a reliability score of 0.72 through fuzzy inference, and select the pure vision mode; Step 7: Plan the path to the stairs to be 12 meters long, generate speed commands (vx=0.5m / s, vy=0, vz=0) through MPC, and convert them into motor control quantities for execution.

[0165] The drone flew along the corridor to the stairs. Upon reaching the 3rd floor, the smoke concentration increased to 0.75, the number of feature points decreased to 89, and the reliability score dropped to 0.25. It switched to IMU-dominated mode, initiating an exploration strategy to lower the flight altitude and increase texture closer to the ground. The number of feature points recovered to 156, and the score rebounded to 0.58. It then switched back to fusion mode. Continuing to ascend to the 4th floor, the air pressure altitude of 14.8 meters matched the visual marker "4th floor," and the fused floor was determined to be the 4th floor. A search path was planned within the rooms on the 4th floor. Semantic segmentation identified 5 rooms, which were entered sequentially for searching. The infrared camera detected a high-temperature human target, determining the trapped person's location as (x=18.5m, y=6.2m, z=12.3m, floor=4). A return path was planned, returning to the 1st floor exit, completing the mission. The total flight time was 8 minutes and 35 seconds, the total trajectory length was 76 meters, the average positioning error was 0.23 meters, and no collision occurred.

[0166] Navigation data for key locations is shown in Table 1:

[0167] Table 1: Navigation data for key locations;

[0168]

[0169] As shown in Table 1, in the 1st and 4th floors with lower smoke concentrations, the system extracted a sufficient number of feature points, achieved a high in-situ matching rate, and a reliability score exceeding 0.7, enabling high-precision positioning using pure vision mode. In the dense smoke area of ​​the 3rd floor, the number of feature points dropped sharply to 89, the in-situ matching rate fell to 31.5%, and the reliability score was only 0.25. The system automatically switched to IMU-dominated mode, and after initiating the exploration strategy, feature points recovered, the reliability score rebounded, and navigation interruption was successfully avoided. The multimodal fusion floor identification accuracy reached 96%, higher than that of a single sensor.

[0170] The contributions of different modes to localization are shown in Table 2:

[0171] Table 2: Contribution of different modes to localization;

[0172]

[0173] As shown in Table 2, the accuracy of pure visual positioning is 0.18 meters, while pure IMU drifts to 2.35 meters after 1 second. The accuracy improves to 0.12 meters after fusion of visual and IMU data, and further multimodal fusion incorporating barometric pressure and semantic information achieves an accuracy of 0.11 meters. Fusion positioning fully leverages the advantages of each sensor: the IMU provides short-term support when visual perception degrades, visual perception provides correction when the IMU drifts, and floor recognition ensures accurate vertical positioning, achieving robust and reliable navigation performance.

[0174] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A computer vision-based unmanned aerial vehicle (UAV) navigation method, characterized in that, include: The image of the current field of view of the UAV's onboard camera is acquired, and adaptive image enhancement is performed to obtain a multi-scale enhanced image sequence; Scene semantic understanding is performed based on multi-scale enhanced image sequences to obtain semantic label maps; semantic perception feature extraction is performed to obtain feature descriptor subsets; Feature matching and outlier removal are performed based on the feature descriptor subset to obtain the final set of matching point pairs; the essential matrix is ​​calculated based on the final set of matching point pairs using the RANSAC algorithm. Singular value decomposition is performed on the essential matrix; floor identification is performed based on the semantic label map; multi-sensor fusion localization is performed by combining the final matching point set to obtain the fusion optimized pose and root mean square value of reprojection error. Based on the fusion-optimized pose and the final matching point pair set, triangulation is performed. Combined with the semantic label map, plane segmentation, structural constraint extraction and map regularization are performed. A multi-layer map management strategy is adopted to obtain a complete multi-layer semantic map. Visual navigation is evaluated based on the root mean square value of reprojection error. The navigation mode is determined based on the evaluation results. Exploration control commands are generated based on the navigation mode to obtain the exploration control vector. Based on the fusion optimization of pose and the complete multi-layer semantic map, a cross-floor path sequence is generated and the trajectory is optimized in real time. The control is fused by combining navigation mode identifiers and exploration control vectors to obtain the final control command.

2. The UAV navigation method based on computer vision according to claim 1, characterized in that, The current field-of-view images acquired by the UAV-borne camera include the visible light RGB images and infrared thermal images of the current field of view of the UAV-borne visible light binocular camera and infrared thermal imaging camera. Based on visible light RGB images and infrared thermal images, synchronous acquisition is performed at a fixed frame rate, and the original visible light image sequence and the original infrared image sequence are obtained by sorting by timestamp. Adaptive image enhancement is performed based on the original visible light image sequence and the original infrared image sequence. The adaptive image enhancement includes smoke concentration estimation, multimodal fusion, concentration adaptive defogging, contrast enhancement, and multiscale decomposition to obtain the multiscale enhanced image sequence.

3. The UAV navigation method based on computer vision according to claim 2, characterized in that, The smoke concentration estimation includes: Based on the original visible light image, the dark channel prior analysis method is used to perform local window scanning on the original visible light image, calculate the minimum pixel value of the three color channels of RGB in each local window, and statistically analyze the distribution characteristics of the minimum pixel value in the whole image. By analyzing the offset of the dark channel statistical histogram, the initial value of smoke concentration in the visible light field of view is estimated, and the visible light smoke concentration parameter is obtained. Based on the original infrared image sequence, the temperature gradient analysis method is used to map the pixel gray values ​​of the original infrared image to temperature values, calculate the temperature gradient field of the original infrared image, and determine the degree of absorption and attenuation of infrared radiation by the smoke by analyzing the spatial distribution and attenuation characteristics of the temperature gradient, thereby obtaining the infrared smoke concentration parameter. Based on visible light smoke concentration parameters and infrared smoke concentration parameters, an exposure adaptive weighted fusion algorithm is used to calculate the average brightness value of the original visible light image. The average brightness value is compared with a preset overexposure threshold. The fusion weight of the visible light mode is calculated by the Sigmoid function. The infrared mode weight is subtracted from the visible light weight. The smoke concentration parameters of the two modes are fused using a weighted average method to obtain the fused smoke concentration estimate.

4. The UAV navigation method based on computer vision according to claim 1, characterized in that, The scene semantic understanding based on multi-scale enhanced image sequences employs a semantic segmentation network, and the pre-training process of the semantic segmentation loss function is as follows: The total number of pixels in the original enhanced image in the multi-scale enhanced image sequence is counted and denoted as the total number of pixels; Determine the total number of categories in the semantic segmentation task; For each pixel location, obtain the actual semantic category label; For each pixel location, obtain the probability value predicted by the semantic segmentation network that this pixel belongs to each category; For each category of each pixel, calculate the cross-entropy between the true label and the predicted probability, which is to multiply the logarithm of the predicted probability by the true label; The cross-entropy values ​​of all categories for all pixels are summed, and a negative sign is added before the summation to obtain the negative log-likelihood loss. The negative log-likelihood loss is then divided by the total number of pixels to obtain the average semantic segmentation loss value.

5. The UAV navigation method based on computer vision according to claim 1, characterized in that, The semantic-aware feature extraction includes semantic reliability assessment, multi-scale feature detection, semantic weighted filtering, and hybrid feature descriptor construction. Multi-scale feature detection involves performing multi-scale feature detection on each layer of the multi-scale enhanced image sequence, specifically including: An improved FAST corner detection algorithm is used to calculate the gradient magnitude map of each image layer, and the gradient mean and standard deviation of the neighborhood of each image pixel are statistically analyzed. The corner detection threshold is adaptively set according to the local gradient statistical characteristics. The corner detection threshold is set by adding the gradient mean to the adjustment coefficient and multiplying the gradient standard deviation. The adjustment coefficient is dynamically set according to the smoke concentration. By comparing the grayscale differences of each candidate point in each layer of the image, the initial set of corner points for each layer of the image is obtained.

6. The UAV navigation method based on computer vision according to claim 5, characterized in that, The semantic reliability assessment, which involves removing corner points of low-weight regions and retaining feature points of high-reliability semantic regions based on the semantic reliability weight map, is specifically implemented as follows: Based on the semantic label graph, a semantic category reliability assessment method is adopted, and different weights are assigned according to the stability and reliability of different semantic categories in visual navigation; A semantic reliability weight map is obtained by assigning corresponding weights to each pixel position in the semantic label map according to the category. Based on the multi-scale fusion of corner point set and semantic reliability weight map, a semantic weighted filtering strategy is adopted. The weight value of each corner point position in the weight map is read, a weight threshold is set and corner points with weights below the threshold are removed, and corner points in high-reliability semantic regions are retained first, resulting in a semantically filtered feature point set.

7. The UAV navigation method based on computer vision according to claim 1, characterized in that, The steps for floor identification include: Based on semantic label graphs, a deep learning object detection method is used to identify floor signage information. A pre-trained lightweight object detection network is used to detect key floor targets on the semantic label graph, including floor signs, stair signs, and elevator signs. Image cropping is performed on the key target areas of the detected floors. Optical character recognition (OCR) technology is used to identify the floor numbers on the signs. The numerical information is extracted as the visual floor recognition result to obtain the visual floor estimate. Based on the current barometric altitude value measured by the barometric altimeter on the drone, and combined with the historical barometric altitude sequence, a sliding window height change detection method is used to determine the floor change. The difference between the current altitude and the initial altitude is calculated, and then divided by the standard floor height and rounded to obtain the floor change amount. The barometric floor estimate is obtained by adding the floor change amount to the initial floor level. Based on the visual floor estimate, the air pressure floor estimate, and the floor status at the previous moment, the posterior probability distribution of the current floor is calculated using the Bayesian fusion method. A likelihood model is established to evaluate the probability of observing the current measurement under a certain floor condition. Combined with the prior probability of the floor at the previous moment, the posterior probability of each floor is calculated according to the Bayesian formula. The floor with the highest posterior probability is selected as the fused floor judgment result to obtain the current floor identifier.

8. The UAV navigation method based on computer vision according to claim 1, characterized in that, The specific implementation of the structural constraint extraction is as follows: Based on the architectural scene, the planar set is used to extract the geometric constraints of the building structure using a planar relationship analysis algorithm, and to calculate the angle between the normal vectors of any two planes and the distance between their center points. The angle between the normal vectors of two planes determines the type of plane pair. The types of plane pairs include parallel plane pairs and perpendicular plane pairs. The perpendicular relationship between the wall plane and the ground plane and the right angle relationship between adjacent walls are detected, and these geometric constraints are expressed as constraint equations to obtain the set of structural constraints.

9. The UAV navigation method based on computer vision according to claim 1, characterized in that, The steps of the visual navigation evaluation include: Based on the number of feature points in the current frame, the matching inlier rate, the root mean square value of the reprojection error, and the smoke concentration parameter, a fuzzy inference system is used to evaluate the reliability of visual navigation and to fuzzify each index. The number of feature points is divided into three fuzzy sets: small, medium, and large; the matching in-point rate is divided into three fuzzy sets: low, medium, and high; the reprojection error is divided into three fuzzy sets: small, medium, and large; and the smoke concentration is divided into three fuzzy sets: low, medium, and high. Define fuzzy inference rules, calculate the fuzzy distribution of reliability scores based on the fuzzy rules and the fuzzy membership degrees of each index, and obtain the visual reliability score through defuzzification.

10. A computer vision-based unmanned aerial vehicle (UAV) navigation system, characterized in that, A method for performing a computer vision-based UAV navigation method according to any one of claims 1-9, comprising: The image processing module is used to acquire the current field-of-view image of the UAV's onboard camera, perform adaptive image enhancement, and obtain a multi-scale enhanced image sequence; The semantic-aware feature extraction module performs scene semantic understanding based on multi-scale enhanced image sequences, obtains a semantic label map, and performs semantic-aware feature extraction to obtain a feature descriptor subset. The feature matching module performs feature matching and outlier removal based on the feature descriptor subset to obtain the final set of matched point pairs; the essence matrix is ​​calculated based on the final set of matched point pairs using the RANSAC algorithm. The multimodal fusion localization module performs singular value decomposition on the essential matrix; it identifies floors based on the semantic label map, and performs multi-sensor fusion localization by combining the final matching point set to obtain the fusion optimized pose and the root mean square value of the reprojection error; The map building module performs triangulation based on the fusion-optimized pose and the final matching point pair set, combines the semantic label map for plane segmentation, structural constraint extraction and map regularization, and adopts a multi-layer map management strategy to obtain a complete multi-layer semantic map. The visual degradation detection module performs visual navigation evaluation based on the root mean square value of reprojection error, determines the navigation mode based on the evaluation results, generates exploration control commands based on the navigation mode, and obtains the exploration control vector. The intelligent path planning and control module generates cross-floor path sequences based on fused and optimized pose and complete multi-layer semantic maps, and optimizes the trajectory in real time. It combines navigation mode identifiers and exploration control vectors to perform control fusion and obtain the final control command.

Images