A method and system for autonomous inspection positioning and navigation of a UAV based on machine vision

Abstract

The application discloses a kind of unmanned aerial vehicle autonomous inspection positioning navigation method and system based on machine vision, it is related to inspection positioning navigation, the present application includes step one, image acquisition, step two, matching path, step three, inspection target identification and step four, adaptive obstacle avoidance and dynamic waypoint adjustment.Differentiation obstacle avoidance strategy is adopted, dynamic obstacle hovers and waits, static obstacle intelligent re-planning path, solve the flight safety problem caused by many interference factors and environmental dynamic change in complex outdoor inspection scene, widen the applicable scene of unmanned aerial vehicle autonomous inspection.Four, through the difference between actual view and expected view adaptive optimization pose, determine whether unmanned aerial vehicle reaches optimal inspection observation posture, automatically decision-making executes inspection operation or generates optimal navigation waypoint sequence, ensure the accuracy of inspection operation, avoid missing inspection, deviation inspection problem, improve inspection operation efficiency.

Description

Technical Field

[0001] This invention relates to the field of inspection, positioning, and navigation technology, specifically to a machine vision-based autonomous inspection, positioning, and navigation method and system for unmanned aerial vehicles (UAVs). Background Technology

[0002] With the rapid development of UAV visual perception and autonomous navigation technologies, UAV autonomous inspection has gradually replaced manual inspection and become the mainstream solution for infrastructure inspection. Currently, existing UAV inspection positioning and navigation technologies mostly rely on single geometric feature matching or preset fixed routes to complete operations, exhibiting significant technical shortcomings. Therefore, it is necessary to analyze a machine vision-based UAV autonomous inspection positioning and navigation method and system.

[0003] Existing technologies, such as the invention patent application CN105389988B, disclose a multi-drone collaborative intelligent highway inspection system. This system uses multi-rotor drones as carriers and combines aerial photography technology to achieve comprehensive, zero-blind-spot inspection of highways, obtaining multiple traffic condition parameters such as vehicle speed, traffic flow, and road occupancy. It also monitors illegal vehicles and road conditions in real time. Simultaneously, it employs GPS positioning and image processing-integrated navigation technology to display areas of interest at the aerial photography center. Machine vision application control theory enables autonomous drone flight, requiring no human operation and achieving long-distance flight. The system uses multi-rotor drones for coordinated inspection, establishing a ground-based transfer station to achieve full-section drone inspection. Furthermore, color recognition enables precise drone landing, wireless power supply is convenient and efficient, and solar panels provide energy conservation and environmental friendliness. A drone transfer station protection device maximizes the protection of the drones and wireless charging devices.

[0004] While existing technologies for machine vision-based autonomous UAV inspection, positioning, and navigation methods and systems can meet basic requirements, they also present some potential shortcomings and challenges, specifically in the following aspects: First, existing technologies do not accurately analyze pixel-level semantic features extracted from scenes using lightweight semantic segmentation networks. This leads to decreased accuracy in distinguishing between various semantic scenes such as roads, buildings, inspection targets, and dynamic objects, thus expanding the search range and hindering the avoidance of mismatches in complex scenes. Second, existing technologies do not place sufficient emphasis on analyzing semantic topology maps matched with multi-frame pose trajectory sequences. This increases positioning errors caused by uneven lighting, scene similarity, and image noise, reducing the pose estimation accuracy and positioning stability of UAVs in complex inspection scenarios. Third, existing technologies lack the adoption of differentiated obstacle avoidance strategies, leading to flight safety issues caused by numerous interference factors and dynamic environmental changes in complex outdoor inspection scenarios. This reduces the applicable scenarios for autonomous UAV inspection, affecting the judgment of whether the UAV has reached the optimal inspection observation attitude, impacting the accuracy of inspection operations, leading to missed or biased inspections, and reducing inspection efficiency. Summary of the Invention

[0005] To address the aforementioned technical shortcomings, the present invention aims to provide a machine vision-based autonomous inspection, positioning, and navigation method and system for unmanned aerial vehicles (UAVs).

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides an autonomous inspection, positioning and navigation method for unmanned aerial vehicles based on machine vision, including: step one, image acquisition; step two, path matching; step three, inspection target identification; and step four, adaptive obstacle avoidance and dynamic waypoint adjustment.

[0007] Step 1, Image Acquisition: Acquire real-time image frames of the UAV inspection area, extract features from the image frames, and obtain semantic feature maps and geometric feature description sets.

[0008] Step 2, Path Matching: Based on the obtained semantic feature map and geometric feature description set, semantic-geometric joint matching is performed with the pre-stored inspection area map to obtain frame-level pose estimation. The frame-level pose estimations of multiple consecutive frames are concatenated into a pose trajectory sequence. The pose trajectory sequence is matched with the topological path in the pre-stored semantic topology map. The map coordinates of the corresponding topological nodes on the matching path are used as global position constraints, and the frame-level pose estimation is corrected in reverse to obtain the initial pose estimation.

[0009] Step 3, Inspection Target Identification: Based on the standard observation position, standard observation posture and standard appearance model of the inspected target corresponding to the current inspection task in the pre-stored inspection task list, and combined with the initial pose estimation, the expected semantic view is generated. The semantic feature map is used as the actual semantic view. The difference between the actual semantic view and the expected semantic view is calculated. The difference is set as the target back-optimization pose parameters to obtain the pose estimate after localization correction.

[0010] Step 4, Adaptive obstacle avoidance and dynamic waypoint adjustment: Based on the corrected pose estimation and semantic feature map, locate the map coordinates of the inspection target. According to the deviation between the map coordinates of the inspection target and the standard observation position and standard observation attitude, decide whether to execute the current inspection task or generate a navigation waypoint sequence, and perform semantic adaptive obstacle avoidance and dynamic waypoint adjustment.

[0011] Furthermore, the specific analysis method for extracting features from the image frames to obtain semantic feature maps and geometric feature description sets is as follows: image sequences of the UAV inspection area are acquired in real time by an airborne camera, each frame of the image is denoised and histogram equalized, and then uniformly scaled to a preset size to obtain preprocessed image frames. The preprocessed image frames are then input into a lightweight semantic segmentation network to extract semantic feature maps, wherein the semantic feature maps contain the probability distribution of each pixel belonging to each semantic category.

[0012] Corner features and line segment features are extracted from the preprocessed image frames, and corner descriptor subsets and line segment descriptor subsets are constructed respectively, and then merged into a geometric feature descriptor set.

[0013] Furthermore, based on the obtained semantic feature map and geometric feature description set, semantic-geometric joint matching is performed with the pre-stored inspection area map to obtain frame-level pose estimation, and the frame-level pose estimation of multiple consecutive frames is concatenated into a pose trajectory sequence. The specific analysis method is as follows: read the pre-stored inspection area map, wherein the map contains a map semantic feature map and a map geometric feature description set, compare the semantic consistency of the semantic feature map with the map semantic feature map to obtain a semantic similarity score, and select the area with the score exceeding a preset threshold as the semantic matching area.

[0014] Within the semantic matching region, feature matching is performed between the geometric feature description set and the map geometric feature description set to obtain a candidate matching pair set. Random sampling consistency screening is performed on the candidate matching pair set to remove mismatched pairs, resulting in a refined matching pair set. The rotation matrix and translation vector of the current image frame with respect to the map coordinate system are calculated to obtain the frame-level pose estimation.

[0015] Obtain frame-level pose estimates for the most recent N consecutive frames of images, concatenate them in chronological order to form a pose trajectory sequence, where N is the preset trajectory window length.

[0016] Furthermore, the sequence matching of the pose trajectory sequence with the topological path in the pre-stored semantic topology graph, using the map coordinates of the corresponding topological nodes on the matching path as global position constraints, and back-correcting the frame-level pose estimation to obtain the initial pose estimation, is specifically analyzed as follows: the pre-stored semantic topology graph is read, wherein the semantic topology graph uses semantically significant regions in the inspection area map as topological nodes, and the traversable paths between semantically significant regions as topological edges. Each topological node is associated with the map coordinates and semantic fingerprint vector of the region, wherein the semantic fingerprint vector is a histogram of the distribution of various semantic categories within a preset radius around the region, and each topological edge is composed of a group of continuous topological nodes in the traversal order to form a topological path.

[0017] The pose trajectory sequence is matched with each topological path in the semantic topology graph. The cumulative matching cost between the pose trajectory sequence and each topological path is calculated. The topological path with the minimum cumulative matching cost is selected as the matching path. The map coordinates of the topological nodes at the corresponding positions on the matching path are used as global position constraints.

[0018] Calculate the position deviation between the global position constraint and the corresponding frame-level pose estimation in the pose trajectory sequence, and use the position deviation as a correction amount to compensate for the frame-level pose estimation of each frame to obtain the initial pose estimation after inverse correction.

[0019] Furthermore, the method for generating the expected semantic view based on the standard observation position, standard observation posture, and standard appearance model of the inspected target corresponding to the current inspection task in the pre-stored inspection task list, and combined with the initial pose estimation, is as follows: obtain the preset inspection task list, and read the standard observation position, standard observation posture, and standard appearance model of the inspected target corresponding to the current inspection task from the inspection task list.

[0020] Based on the initial pose estimation, the imaging parameters of the current camera are determined. Combining the standard observation position and standard observation attitude, the relative spatial relationship between the airborne camera and the inspected target under the current pose is calculated, including the relative position vector and the relative orientation angle.

[0021] Based on relative spatial relationships and a standard appearance model, the standard appearance model is projected onto a two-dimensional imaging plane using a differentiable rendering engine to generate the expected appearance image that should theoretically be captured in the current pose.

[0022] The expected appearance image is input into a lightweight semantic segmentation network, and pixel-by-pixel semantic annotation is performed on the expected appearance image to obtain the expected semantic view. Each pixel in the expected semantic view contains the corresponding semantic category probability distribution.

[0023] Furthermore, the specific analysis method for calculating the difference between the actual semantic view and the expected semantic view is as follows: based on the obtained semantic feature map, the actual semantic view is obtained, wherein each pixel in the actual semantic view contains a probability distribution belonging to each semantic category, and the expected semantic view is obtained, wherein each pixel in the expected semantic view contains a probability distribution belonging to each semantic category.

[0024] For pixels at the same coordinate position in the actual semantic view and the expected semantic view, extract the probability distribution of the semantic category to which the pixel belongs in each of the two semantic views, and calculate the cross-entropy value between the two probability distributions.

[0025] Obtain the pixels at all coordinate positions in the actual semantic view and the expected semantic view. Sum the cross-entropy values ​​calculated at each coordinate position and divide by the total number of pixels in the semantic view to obtain the average cross-entropy loss value.

[0026] Furthermore, the specific analysis method for setting the difference as the target back-optimization pose parameters to obtain the pose estimate after localization correction is as follows: the average cross-entropy loss value is used as the optimization objective function, the rotation matrix and translation vector in the initial pose estimate of the current frame are used as the pose parameters to be optimized, and the gradient values ​​of each angle component in the rotation matrix and each position component in the translation vector of the average cross-entropy loss value are calculated by the backpropagation algorithm.

[0027] Multiply each gradient value by the preset learning rate to obtain the update amount corresponding to each angle component and each position component. Subtract the corresponding update amount from each angle component and each position component to obtain the updated pose parameters.

[0028] The expected semantic view is regenerated based on the updated pose parameters, and the corresponding average cross-entropy loss value is calculated. If the updated average cross-entropy loss value is greater than the preset convergence threshold, the process is repeated until the average cross-entropy loss value is less than or equal to the convergence threshold. The pose parameters obtained from the last execution are then used as the pose estimate after localization correction.

[0029] Furthermore, the specific analysis method for deciding whether to execute the current inspection task or generate a navigation waypoint sequence based on the deviation between the map coordinates of the inspection target and the standard observation position and standard observation attitude is as follows: extract pixel regions with the semantic category of the inspection target from the semantic feature map to obtain the inspection target region, and locate the center point image coordinates of the inspection target in the preprocessed image frame based on the inspection target region.

[0030] After obtaining the pose estimate after positioning correction, and combining it with the intrinsic parameter matrix of the airborne camera, the center point image coordinates are transformed to the map coordinate system through inverse projection transformation to obtain the map coordinates of the inspection target.

[0031] Read the standard observation position and standard observation attitude corresponding to the current inspection task from the inspection task list, calculate the position deviation vector between the map coordinates of the inspection target and the standard observation position, and calculate the attitude deviation angle between the orientation of the current pose and the standard observation attitude. Combine the position deviation vector and the attitude deviation angle into a six-degree-of-freedom pose alignment deviation.

[0032] Determine whether the six-degree-of-freedom pose alignment deviation is less than a preset alignment threshold; if so, determine that the UAV has aligned with the inspection target, generate an inspection task execution command, control the UAV to execute the current inspection task and record the inspection data; if not, based on the six-degree-of-freedom pose alignment deviation and combined with the standard observation position, uniformly interpolate to generate a sequence of navigation waypoints on the straight path from the current pose to the standard observation position. Each navigation waypoint contains the target position coordinates and the target orientation angle.

[0033] Furthermore, the specific analysis method for semantic adaptive obstacle avoidance and dynamic waypoint adjustment is as follows: during flight along the navigation waypoint sequence, the current frame image is acquired in real time by the airborne camera to obtain the second-scale semantic feature map of the current frame. Pixel regions with the semantic category of obstacle are extracted from the second-scale semantic feature map to obtain the obstacle region. The spatial range occupied by the obstacle is calculated based on the obstacle region, and it is detected whether the spatial range of the obstacle interferes with the target path of the next waypoint in the navigation waypoint sequence. If no spatial interference occurs, flight continues along the original navigation waypoint sequence.

[0034] If spatial interference occurs, the semantic category of the obstacle is identified from the second-scale semantic feature map, and it is determined whether the semantic category of the obstacle belongs to the dynamic object category or the static obstacle category.

[0035] If the semantic category of the obstacle belongs to the dynamic object category, a hovering waiting command is generated to control the drone to hover in place until the obstacle is detected to have left the interference area, after which it resumes flight along the original navigation waypoint sequence.

[0036] If the semantic category of the obstacle belongs to the static obstacle category, then based on the current UAV pose and the target waypoint position in the navigation waypoint sequence, the A* algorithm is used to perform local path replanning on the obstacle-occupied grid of the second-scale semantic feature map to generate a detour waypoint sequence, which replaces the interference segment in the original navigation waypoint sequence.

[0037] A second aspect of the present invention provides a system for executing the machine vision-based UAV autonomous inspection, positioning, and navigation method, comprising: an image acquisition module: acquiring real-time image frames of the UAV inspection area, extracting features from the image frames, and obtaining a semantic feature map and a geometric feature description set.

[0038] The matching path module: Based on the obtained semantic feature map and geometric feature description set, it performs semantic-geometric joint matching with the pre-stored inspection area map to obtain frame-level pose estimation. The frame-level pose estimation of multiple consecutive frames is concatenated into a pose trajectory sequence. The pose trajectory sequence is matched with the topological path in the pre-stored semantic topology map. The map coordinates of the corresponding topological nodes on the matching path are used as global position constraints, and the frame-level pose estimation is corrected in reverse to obtain the initial pose estimation.

[0039] Inspection target identification module: Based on the standard observation position, standard observation posture and standard appearance model of the inspected target corresponding to the current inspection task in the pre-stored inspection task list, and combined with the initial pose estimation, the expected semantic view is generated. The semantic feature map is used as the actual semantic view. The difference between the actual semantic view and the expected semantic view is calculated. The difference is set as the target back optimization pose parameters to obtain the pose estimate after localization correction.

[0040] The adaptive obstacle avoidance and dynamic waypoint adjustment module locates the map coordinates of the inspection target based on the corrected pose estimation and semantic feature map. According to the deviation between the map coordinates of the inspection target and the standard observation position and standard observation attitude, it decides to execute the current inspection task or generate a navigation waypoint sequence, and performs semantic adaptive obstacle avoidance and dynamic waypoint adjustment.

[0041] The beneficial effects of this invention are as follows: 1. By extracting pixel-level semantic features of a scene through a lightweight semantic segmentation network, it accurately distinguishes multiple semantic scenes such as roads, buildings, inspection targets, and dynamic objects. It also filters matching regions based on semantic connected components, narrowing the search range and avoiding mismatches in complex scenes. 2. By matching semantic topology maps with multi-frame pose trajectory sequences, it reverses frame-level pose correction using global topology node coordinates, and then combines this with differentiable rendering to generate the expected semantic view. This reduces positioning errors caused by uneven lighting, scene similarity, and image noise, improving the pose estimation accuracy and positioning stability of UAVs in complex inspection scenarios. 3. By adopting differentiated obstacle avoidance strategies—hovering and waiting for dynamic obstacles and intelligently replanning paths for static obstacles—it solves flight safety problems caused by numerous interference factors and dynamic environmental changes in complex outdoor inspection scenarios, broadening the applicable scenarios for autonomous UAV inspections. Fourth, by adaptively optimizing the pose based on the difference between the actual view and the expected view, it can determine whether the UAV has reached the optimal inspection and observation attitude, and automatically decide whether to execute the inspection operation or generate the optimal navigation waypoint sequence, which ensures the accuracy of the inspection operation, avoids the problems of missed inspections and biased inspections, and improves the efficiency of the inspection operation. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the implementation steps of the method of the present invention.

[0044] Figure 2 This is a schematic diagram of the system structure connection of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Reference Figure 1 As shown, the present invention provides an autonomous inspection, positioning and navigation method for unmanned aerial vehicles based on machine vision, including: step one, image acquisition; step two, path matching; step three, inspection target identification; and step four, adaptive obstacle avoidance and dynamic waypoint adjustment.

[0047] Step 1, Image Acquisition: Acquire real-time image frames of the UAV inspection area, extract features from the image frames, and obtain semantic feature maps and geometric feature description sets.

[0048] Step 2, Path Matching: Based on the obtained semantic feature map and geometric feature description set, semantic-geometric joint matching is performed with the pre-stored inspection area map to obtain frame-level pose estimation. The frame-level pose estimations of multiple consecutive frames are concatenated into a pose trajectory sequence. The pose trajectory sequence is matched with the topological path in the pre-stored semantic topology map. The map coordinates of the corresponding topological nodes on the matching path are used as global position constraints, and the frame-level pose estimation is corrected in reverse to obtain the initial pose estimation.

[0049] Step 3, Inspection Target Identification: Based on the standard observation position, standard observation posture and standard appearance model of the inspected target corresponding to the current inspection task in the pre-stored inspection task list, and combined with the initial pose estimation, the expected semantic view is generated. The semantic feature map is used as the actual semantic view. The difference between the actual semantic view and the expected semantic view is calculated. The difference is set as the target back-optimization pose parameters to obtain the pose estimate after localization correction.

[0050] Step 4, Adaptive obstacle avoidance and dynamic waypoint adjustment: Based on the corrected pose estimation and semantic feature map, locate the map coordinates of the inspection target. According to the deviation between the map coordinates of the inspection target and the standard observation position and standard observation attitude, decide whether to execute the current inspection task or generate a navigation waypoint sequence, and perform semantic adaptive obstacle avoidance and dynamic waypoint adjustment.

[0051] In the above embodiments, the specific analysis method for extracting features from image frames to obtain semantic feature maps and geometric feature description sets is as follows: image sequences of the UAV inspection area are acquired in real time by an airborne camera, each frame of the image is denoised and histogram equalized, and uniformly scaled to a preset size to obtain preprocessed image frames. The preprocessed image frames are then input into a lightweight semantic segmentation network to extract semantic feature maps, wherein the semantic feature maps contain the probability distribution of each pixel belonging to each semantic category.

[0052] Corner features and line segment features are extracted from the preprocessed image frames, and corner descriptor subsets and line segment descriptor subsets are constructed respectively, and then merged into a geometric feature descriptor set.

[0053] It should be noted that denoising the original acquired image frames can eliminate noise interference introduced by the airborne camera sensor, histogram equalization can enhance the image contrast, so that subsequent feature extraction is not affected by uneven lighting, and scaling to a preset size ensures that the input size of the semantic segmentation network and the geometric feature extraction algorithm is consistent, thus guaranteeing processing efficiency and data alignment.

[0054] It should be noted that the lightweight semantic segmentation network adopts an encoder-decoder structure. The encoder part extracts multi-scale semantic features by downsampling layer by layer through depthwise separable convolutions, and the decoder part restores the original input resolution through upsampling and skip connections. Finally, it outputs the probability distribution of each pixel belonging to each semantic category. The semantic categories include roads, buildings, vegetation, water surfaces, dynamic objects, and inspection target categories. The inspection target categories are predefined according to different inspection tasks, including power towers, pipeline valves, bridge supports, etc.

[0055] It should be noted that the specific analysis process for corner features and line segment features is as follows: Corner features use an accelerated segmentation test algorithm to extract pixels with drastic grayscale changes in the image as corners, and construct a description vector for each corner based on binary robust independent basic feature descriptors; line segment features use a line segment detector algorithm to extract straight line segments with continuous edges in the image, and construct a description vector for each line segment based on the line segment angle, length, and neighborhood gradient distribution; the set of corner descriptors and the set of line segment descriptors are merged into a set of geometric feature descriptions for use in geometric feature matching in subsequent steps.

[0056] It should be noted that the basic principle of the accelerated segmentation test algorithm is as follows: taking the pixel to be detected as the center and the preset radius as the search circle, the brightness value of the pixel to be detected is compared with the brightness value of each pixel on the search circle. If there are a preset number of consecutive pixels on the search circle whose brightness values ​​differ from the brightness value of the pixel to be detected by a preset brightness threshold, then the pixel to be detected is determined to be a corner point.

[0057] It should be noted that the specific execution process of the accelerated segmentation test algorithm on the preprocessed image frame is as follows: First, a rapid pre-screening of corner candidate pixels is performed on each pixel in the preprocessed image frame. Taking the current pixel as the detection point, and using pixels on a search circle with a preset radius as an example, the four pixels located directly above, below, to the left, and to the right of the search circle are checked. The number of pixels whose brightness value is greater than the brightness value of the detection pixel plus a preset brightness threshold, and the number of pixels whose brightness value is less than the brightness value of the detection pixel minus the preset brightness threshold, are counted. If at least a preset number of pixels among the above four pixels meet the brightness difference condition, the current pixel is marked as a candidate corner; otherwise, the current pixel is directly excluded, thus quickly filtering a large number of non-corner pixels and reducing subsequent computation. Second, a complete circumferential detection is performed on the pixels marked as candidate corners. The process involves three steps: First, obtain all pixels on a search circle with a preset radius corresponding to the candidate corner point. Count the number of pixels whose brightness is greater than the brightness value of the pixel to be detected plus a preset brightness threshold, and the number of pixels whose brightness is less than the brightness value of the pixel to be detected minus the preset brightness threshold. If the larger of these two consecutive pixel counts is greater than a preset corner point determination threshold, the current candidate corner point is determined to be a corner point. Second, perform non-maximum suppression on all detected corner points to eliminate corner point clustering. For each corner point, calculate its corner response value, which is the sum of the absolute values ​​of the brightness differences between all pixels on the search circle and the pixel to be detected. A larger corner response value indicates a higher degree of significance for the corner point. Compare the corner response value of each corner point with the corner response values ​​of other corner points within its preset neighborhood. Only retain the corner point with the largest corner response value and remove other corner points in the neighboring area to obtain the final set of non-repeating corner points.

[0058] It should be noted that the preset radius ranges from one to three pixels, the preset brightness threshold ranges from ten to sixty brightness levels, the preset quantity ranges from nine to twelve pixels, and the preset corner detection threshold ranges from nine to sixteen pixels. The above parameters are calibrated and set according to the image resolution and lighting conditions of the actual scene.

[0059] It should be noted that for each detected corner point, a corner point description vector is constructed using a binary robust independent basic feature descriptor. The construction process is as follows: with the corner point as the center, several sets of pixel pairs are selected within an image block of a preset size according to a preset sampling point pair pattern. For each set of pixel pairs, the brightness values ​​of the two pixels are compared. If the brightness value of the first pixel is greater than that of the second pixel, the comparison result is recorded as a value of one; otherwise, it is recorded as a value of zero. The comparison results of each set are arranged in order to obtain a string of binary codes, which serves as the corner point description vector. The corner point description vector has rotation invariance and scale invariance, and is calculated quickly with a small storage space.

[0060] It should be noted that the basic principle of the line segment detector algorithm is as follows: based on the consistency of the image gradient direction, pixels with similar gradient directions and spatial proximity are merged into line support regions, and then a corresponding straight line segment is fitted to each line support region, and the significance of the straight line segment is verified.

[0061] It should be noted that the specific execution process of the line segment detector algorithm on the preprocessed image frame is as follows: First, calculate the gradient direction and gradient magnitude of each pixel in the preprocessed image frame. The gradient direction is the direction in which the pixel's brightness value changes the fastest, and the gradient magnitude is the magnitude of the change in the pixel's brightness value. Pixels with larger gradient magnitudes are more likely to belong to edge regions, while pixels with smaller gradient magnitudes are more likely to belong to flat regions. Second, exclude pixels with gradient magnitudes less than a preset gradient threshold, and only perform subsequent processing on pixels with gradient magnitudes greater than or equal to the preset gradient threshold. The preset gradient threshold is used to filter noise and flat regions, reducing invalid calculations. The third step involves selecting unmarked pixels from the remaining pixels as seed pixels. Starting from these seed pixels, neighboring pixels whose gradient directions are less than a preset threshold are recursively merged into the same line support region until no new pixels are added. After constructing one line support region, the next unmarked seed pixel is selected to continue constructing the next line support region until all pixels satisfying the gradient magnitude condition are assigned to the corresponding line support region. The fourth step involves fitting a straight line segment for each line support region using the least squares method. The geometric center of all pixels in the line support region is calculated as the center point of the straight line segment. The principal direction vector of the line support region is calculated; this principal direction vector is the direction vector obtained by rotating the statistical average direction of the gradient directions of all pixels by 90 degrees. The center position and direction of the straight line segment are determined based on the center point and the principal direction vector. The projection range of all pixels in the line support region onto the principal direction vector is taken as the length of the straight line segment, thus obtaining the fitted straight line segment. The fifth step involves validating each fitted straight line segment. Calculate the saliency score of the line segment. The saliency score is the proportion of the number of pixels in the line support region that are in the same direction as the fitted line segment to the total number of pixels in the line support region. If the saliency score is greater than the preset saliency threshold, the fitted line segment is determined to be valid and retained as a line segment feature; otherwise, the fitted line segment is determined to be invalid and discarded.

[0062] It should be noted that the process of constructing a line segment description vector for each detected line segment feature is as follows: obtain the coordinates of the two endpoints of the line segment feature, calculate the length, angle, and center point position of the line segment feature; based on the center point of the line segment feature, expand the width to both sides along the normal direction of the line segment feature to form a rectangular image block; divide the rectangular image block into a predetermined number of sub-regions, calculate the gradient direction histogram of pixels in each sub-region, and concatenate the gradient direction histograms of each sub-region in sequence to obtain the line segment description vector of the line segment feature; the line segment description vector has rotation invariance and illumination invariance.

[0063] It should be noted that the set of corner point description vectors and the set of line segment description vectors are merged to form a set of geometric feature descriptions, which is used for subsequent geometric feature matching.

[0064] In the above embodiments, the semantic feature map and geometric feature description set obtained are used to perform semantic-geometric joint matching with the pre-stored inspection area map to obtain frame-level pose estimation, and the frame-level pose estimation of multiple consecutive frames is concatenated into a pose trajectory sequence. The specific analysis method is as follows: read the pre-stored inspection area map, wherein the map contains a map semantic feature map and a map geometric feature description set, compare the semantic consistency between the semantic feature map and the map semantic feature map to obtain a semantic similarity score, and select the area with the score exceeding a preset threshold as the semantic matching area.

[0065] Within the semantic matching region, feature matching is performed between the geometric feature description set and the map geometric feature description set to obtain a candidate matching pair set. Random sampling consistency screening is performed on the candidate matching pair set to remove mismatched pairs, resulting in a refined matching pair set. The rotation matrix and translation vector of the current image frame with respect to the map coordinate system are calculated to obtain the frame-level pose estimation.

[0066] Obtain frame-level pose estimates for the most recent N consecutive frames of images, concatenate them in chronological order to form a pose trajectory sequence, where N is the preset trajectory window length.

[0067] It should be noted that the pre-stored inspection area map is a high-precision map pre-built by the UAV before the inspection mission is executed. The map semantic feature map is a semantically labeled image generated after offline semantic segmentation of the inspection area, and the map geometric feature description set is a feature database built after extracting corner point and line segment features of the inspection area.

[0068] It should be noted that the semantic consistency comparison method is as follows: First, the current semantic feature map and the map semantic feature map are obtained. The current semantic feature map and the map semantic feature map are the same size, each containing a preset number of pixels at a preset height and width. For each pixel at the same coordinate position in the current semantic feature map and the map semantic feature map, the semantic category of the pixel at that position is read from both the current semantic feature map and the map semantic feature map. It is then determined whether the two semantic categories are equal: if they are equal, the comparison result at that position is recorded as a value of one; if they are not equal, it is recorded as a value of zero. The values ​​of one from all position comparison results are summed to obtain the total number of consistent pixels. The total number of consistent pixels is divided by the total number of pixels in the semantic feature map to obtain the semantic similarity score. The semantic similarity score ranges from zero to one. The closer the score is to one, the stronger the semantic consistency between the current semantic feature map and the map semantic feature map. The region with a semantic similarity score exceeding a preset threshold is determined as the semantic matching region and used as the search range for subsequent geometric feature matching.

[0069] It should be noted that the process of defining the semantic matching region in the map semantic feature map is as follows: The map semantic feature map is analyzed for connectivity based on the semantic categories of pixels. All spatially connected pixel regions with consistent semantic categories are extracted as candidate semantic connectivity regions. Each candidate semantic connectivity region corresponds to a local region with continuous semantic attributes in the map, such as a complete road area, a building roof area, or a continuous water surface area. Each candidate semantic connectivity region has a clearly defined set of spatial boundary coordinates. The semantic consistency of the current frame's semantic feature map is compared with the map semantic feature map within the range corresponding to each candidate semantic connectivity region. The semantic similarity score between the current frame's semantic feature map and each candidate semantic connectivity region is calculated. The semantic consistency comparison operation is limited to the bounding box of the candidate semantic connectivity region; that is, only the pixel regions in the current frame's semantic feature map corresponding to the spatial location of the candidate semantic connectivity region are calculated. Candidate semantic connectivity regions with semantic similarity scores exceeding a preset threshold are selected as semantic matching regions. If multiple candidate semantic connectivity regions have scores exceeding the preset threshold, the spatial ranges of these candidate semantic connectivity regions are merged to form the final semantic matching region.

[0070] It should be noted that the process of defining the semantic matching region is as follows: Suppose the map semantic feature map contains a rectangular building roof area with the semantic category of building and an irregular lake area with the semantic category of water surface, and the semantic feature map of the current frame contains part of the building. First, through connected component analysis, the building roof connected component and the lake connected component are extracted as two candidate semantic connected components. The semantic similarity scores between the semantic feature map of the current frame and the map semantic feature map within the building roof connected component, and between the semantic feature map of the current frame and the map semantic feature map within the lake connected component are calculated. Assuming the semantic similarity score of the building roof connected component is 0.85 and the semantic similarity score of the lake connected component is 0.12, and the preset threshold is set to 0.70, then only the building roof connected component is used as the semantic matching region. Subsequent geometric feature matching is limited to the spatial range of the building roof connected component, thereby significantly narrowing the matching search range and reducing the occurrence of false matches. The preset threshold ranges from 0.50 to 0.90, and the specific value is set according to the semantic complexity of the inspection scenario: in areas with rich semantic categories and high scene complexity, a higher preset threshold is used to ensure the reliability of the matching region; in areas with simple semantic categories and simple scenes, the preset threshold can be appropriately reduced to ensure sufficient matching region coverage.

[0071] It should be noted that the feature matching uses the nearest neighbor distance ratio algorithm. For each feature descriptor in the geometric feature description set, the algorithm finds the two closest and second closest matching feature descriptors in the map geometric feature description set, calculates the ratio of the nearest distance to the second closest distance, and if the ratio is less than the preset matching threshold, the current feature descriptor and the nearest neighbor feature descriptor are determined to be a matching pair.

[0072] It should be noted that the random sampling consistency screening process is as follows: a preset number of matching pairs are randomly selected from the candidate matching pair set to calculate the initial transformation matrix. The remaining matching pairs are substituted into the initial transformation matrix to calculate the projection error. The number of matching pairs with an error less than a preset error threshold is taken as the number of inliers. After repeating the above process for a preset number of iterations, the matching pair set with the largest number of inliers is selected as the refined matching pair set. The transformation matrix is ​​then recalculated based on this set to obtain the frame-level pose estimation, including the rotation matrix and translation vector.

[0073] It should be noted that each matching pair in the refined matching pair set contains two elements: the first element is the pixel coordinates of the two-dimensional image feature point in the current image frame, and the second element is the map coordinates of the three-dimensional spatial point corresponding to the two-dimensional image feature point in the map coordinate system; the two-dimensional image feature point comes from the endpoints of the extracted corner feature or line segment feature, and the three-dimensional spatial point comes from the spatial point in the map geometric feature description set in the pre-stored map that matches the feature point.

[0074] It should be noted that the rotation matrix and translation vector of the current image frame relative to the map coordinate system are calculated using the perspective n-point algorithm and based on the geometric constraints of the pinhole camera imaging model. This is achieved by utilizing the known 2D-3D correspondences within the refined matching pair set to solve for the rotation matrix and translation vector of the camera coordinate system relative to the map coordinate system. Specifically, a predetermined number of matching pairs are randomly selected from the refined matching pair set to form a minimum solution subset. Using the 2D-3D correspondences within this minimum solution subset, a system of linear equations is constructed using the direct linear transformation algorithm. Singular value decomposition is then performed on this system of equations to obtain the initial solutions for the rotation matrix and translation vector. The core principle of the direct linear transformation algorithm is as follows: The projection equation of the camera imaging model is rewritten as a homogeneous linear equation. The nine unknowns of the rotation matrix and the three unknowns of the translation vector, totaling twelve unknown parameters, are used as the variables to be solved. Each matching pair provides two linear constraint equations, thus requiring six matching pairs to solve all twelve unknown parameters. However, due to the orthogonal constraint of the rotation matrix (i.e., the three column vectors of the rotation matrix are mutually orthogonal and have a magnitude of 1), the degrees of freedom are reduced to six in actual solution, using three non-collinear matching pairs. All remaining matching pairs in the refined matching pair set are substituted into the projection equation corresponding to the initial solution. The projection coordinates of each 3D space point projected onto the 2D imaging plane according to the initial solution are calculated. The reprojection error between the projection coordinates and the pixel coordinates of the corresponding 2D image feature points is calculated. The number of matching pairs with a reprojection error less than a preset reprojection error threshold is counted as the number of interior points in the initial solution. This process is repeated for a preset number of iterations, and the solution with the most interior points is selected as the optimal solution. Using the optimal solution as the initial value and minimizing the sum of squared reprojection errors of all inlier matching pairs as the objective, the Levenberg-Marquardt algorithm is used to perform nonlinear optimization on the rotation matrix and translation vector to obtain the final optimized rotation matrix and translation vector, which serves as the frame-level pose estimate of the current image frame relative to the map coordinate system.

[0075] It should be noted that the reprojection error is calculated as follows: multiply the map coordinates of the 3D spatial point by the estimated rotation matrix, and add the estimated translation vector to obtain the 3D coordinates of the 3D spatial point in the camera coordinate system; divide the X-axis and Y-axis components of the 3D coordinates in the camera coordinate system by the Z-axis component to obtain the normalized coordinates; multiply the normalized coordinates by the camera intrinsic parameter matrix to obtain the projected coordinates of the 3D spatial point on the 2D imaging plane; calculate the Euclidean distance between the projected coordinates and the pixel coordinates of the corresponding 2D image feature point of the 3D spatial point, where the Euclidean distance is the reprojection error.

[0076] It should be noted that the preset trajectory window length N ranges from 5 to 15 frames. The specific value is dynamically adjusted based on the drone's flight speed V and image acquisition frame rate F. The dynamic adjustment rule is as follows: divide the preset target space coverage length S by the flight speed V to obtain the time required to cover the target space. Multiply the time by the frame rate F and round up to obtain the basic window length N_b. Record the trajectory window length of the previous frame as N_p. Calculate the difference between N_b and N_p. If the absolute value of the difference is less than or equal to the preset jump threshold, then N is set to N_p. Otherwise, N is N_p plus or minus the jump threshold, and N is clamped within the range of 5 to 15; the target space coverage length S is 5 to 20 meters, preset according to the type of inspection scenario. For example, in scenarios where inspection targets are dense and frequent turning and changing direction are required, such as densely built areas, S takes a smaller value, such as 5 meters, to shorten the trajectory window length and improve the sensitivity of positioning response. In scenarios where inspection targets are sparse and the flight path is straight, such as long-distance pipeline inspection, S takes a larger value, such as 20 meters, to increase the trajectory window length and improve the robustness of sequence matching.

[0077] In the above embodiment, the step of performing sequence matching between the pose trajectory sequence and the topological path in the pre-stored semantic topology graph, using the map coordinates of the corresponding topological nodes on the matching path as global position constraints, and back-correcting the frame-level pose estimation to obtain the initial pose estimation, is specifically analyzed as follows: reading the pre-stored semantic topology graph, wherein the semantic topology graph uses semantically significant regions in the inspection area map as topological nodes, and the traversable paths between semantically significant regions as topological edges. Each topological node is associated with the map coordinates and semantic fingerprint vector of the region, wherein the semantic fingerprint vector is a histogram of the distribution of various semantic categories within a preset radius around the region, and each topological edge is composed of a group of continuous topological nodes in the traversal order to form a topological path.

[0078] The pose trajectory sequence is matched with each topological path in the semantic topology graph. The cumulative matching cost between the pose trajectory sequence and each topological path is calculated. The topological path with the minimum cumulative matching cost is selected as the matching path. The map coordinates of the topological nodes at the corresponding positions on the matching path are used as global position constraints.

[0079] Calculate the position deviation between the global position constraint and the corresponding frame-level pose estimation in the pose trajectory sequence, and use the position deviation as a correction amount to compensate for the frame-level pose estimation of each frame to obtain the initial pose estimation after inverse correction.

[0080] It should be noted that semantically salient regions refer to regions with unique semantic-geometric joint features in the map semantic feature map, including road intersections, building corners, isolated trees, and tower bases. The semantic category distribution of regions with semantic-geometric joint features has significant distinguishability and spatial uniqueness, and can be used as landmark nodes for topological positioning. The semantic fingerprint vector is represented by a histogram of semantic category distribution within a preset radius, which can describe the semantic environment features around the region. The value of the preset radius is set according to the scale of the inspection scenario.

[0081] It should be noted that the sequence matching uses an improved dynamic time warping algorithm. The cumulative matching cost is calculated as follows: for each frame in the pose trajectory sequence and each topological node in the topological path, the matching cost between the two is calculated. The matching cost is composed of a weighted sum of spatial distance cost and semantic dissimilarity cost, as follows: calculate the Euclidean distance between the position coordinates of the frame-level pose estimation of the current frame in the pose trajectory sequence and the map coordinates of the topological node as the spatial distance cost. Calculate the cosine similarity between the semantic fingerprint vector corresponding to the current frame and the semantic fingerprint vector associated with the topological node. Subtract the cosine similarity from one to obtain the semantic dissimilarity cost. Multiply the spatial distance cost by a preset spatial distance cost weight, and add the semantic dissimilarity cost multiplied by a preset semantic dissimilarity weight to obtain the matching cost between the current frame and the current topological node.

[0082] It should be noted that after calculating the matching cost between all frames and all topology nodes, the cumulative cost minimization strategy in the dynamic time warping algorithm is adopted. The optimal alignment path between the start and end points of the pose trajectory sequence and the start and end points of the topology path is found through dynamic programming. The cumulative matching cost corresponding to the optimal alignment path is the cumulative matching cost between the topology path and the pose trajectory sequence.

[0083] It should be noted that the specific operation of the reverse correction is as follows: obtain the map coordinates of the topology nodes corresponding to each frame in the pose trajectory sequence after matching with the optimal alignment path from the matching path; take the difference between the map coordinates of the topology nodes and the position coordinates of the frame-level pose estimation of the corresponding frame as the position correction amount of the frame; superimpose the position correction amount onto the translation vector of the frame-level pose estimation of the corresponding frame, and adjust the orientation component in the rotation matrix according to the direction of the position correction amount to obtain the initial pose estimation after reverse correction.

[0084] In the above embodiment, the specific analysis method for generating the expected semantic view based on the standard observation position, standard observation posture and standard appearance model of the inspected target corresponding to the current inspection task in the pre-stored inspection task list, and combined with the initial pose estimation, is as follows: obtain the preset inspection task list, and read the standard observation position, standard observation posture and standard appearance model of the inspected target corresponding to the current inspection task from the inspection task list.

[0085] Based on the initial pose estimation, the imaging parameters of the current camera are determined. Combining the standard observation position and standard observation attitude, the relative spatial relationship between the airborne camera and the inspected target under the current pose is calculated, including the relative position vector and the relative orientation angle.

[0086] Based on relative spatial relationships and a standard appearance model, the standard appearance model is projected onto a two-dimensional imaging plane using a differentiable rendering engine to generate the expected appearance image that should theoretically be captured in the current pose.

[0087] The expected appearance image is input into a lightweight semantic segmentation network, and pixel-by-pixel semantic annotation is performed on the expected appearance image to obtain the expected semantic view. Each pixel in the expected semantic view contains the corresponding semantic category probability distribution.

[0088] It should be noted that the inspection task list is a pre-planned task list before the UAV performs inspection tasks. Each inspection task record includes: task number, identification information of the inspected target, the standard observation position of the inspected target in the map coordinate system, that is, the X, Y, Z three-dimensional coordinates of the ideal shooting position, the standard observation attitude, that is, the camera's pitch angle, yaw angle and roll angle at the ideal shooting time, and the standard appearance model of the inspected target. The standard appearance model is the three-dimensional geometric model of the inspected target, which includes the shape, texture and semantic label information of the target surface, and is used to generate realistic expected images during the differentiable rendering process.

[0089] It should be noted that the calculation process of the relative spatial relationship is as follows: a local coordinate system of the target is established with the standard observation position of the inspected target as the origin. The camera position coordinates in the initial pose estimation are transformed into the local coordinate system of the target to obtain the relative position vector. The relative position vector includes the direction component and the distance component from the inspected target to the camera. At the same time, the camera orientation in the initial pose estimation is compared with the standard observation attitude to obtain the relative orientation angle. The relative orientation angle includes pitch angle deviation, yaw angle deviation and roll angle deviation.

[0090] It should be noted that the differentiable rendering engine is implemented based on a 3D graphics rendering pipeline, specifically including: setting the intrinsic and extrinsic parameters of the virtual camera according to the relative position vector and relative orientation angle. The intrinsic parameter matrix of the virtual camera is consistent with the real intrinsic parameter matrix of the airborne camera. The 3D mesh vertices of the standard appearance model are transformed from the world coordinate system to the camera coordinate system according to the extrinsic parameter matrix, and then projected onto the 2D imaging plane through the intrinsic parameter matrix. The color value and depth value of each projected pixel are calculated to generate the expected appearance image. The differentiable rendering engine supports gradient backpropagation, and the pose parameters can be optimized backward through cross-entropy loss in subsequent steps.

[0091] It should be noted that the expected appearance image is input into a lightweight semantic segmentation network to obtain the expected semantic view. Each pixel in the expected semantic view contains the probability distribution of each semantic category. Among them, the semantic category of the pixel region where the inspected target is located is the inspected target category, and the semantic category of the background region is automatically filled according to the semantic labels in the standard appearance model. Thus, the expected semantic view fully describes the semantic scene layout that should theoretically be seen under the current initial pose estimation.

[0092] It should be noted that the computational flow of the differentiable rendering process is as follows: Based on the camera position coordinates and orientation angle in the current initial pose estimation, combined with the standard observation position and standard observation attitude, the relative position vector and relative orientation angle of the camera relative to the inspected target are calculated. The projection parameters of the virtual camera are set according to the relative position vector and relative orientation angle. The 3D geometric vertex data of the standard appearance model are transformed from the world coordinate system to the camera coordinate system and projected from the camera coordinate system to the image coordinate system according to the set projection parameters, obtaining the pixel coordinates of each vertex on the 2D imaging plane. For each pixel position on the imaging plane, the color value corresponding to that pixel is calculated through interpolation, finally generating the expected appearance image. This expected appearance image is input into a semantic segmentation network to obtain the semantic category probability distribution of each pixel, and the output is the expected semantic view.

[0093] In the above embodiments, the specific analysis method for calculating the difference between the actual semantic view and the expected semantic view is as follows: based on the obtained semantic feature map, the actual semantic view is obtained, wherein each pixel in the actual semantic view contains a probability distribution belonging to each semantic category, and the expected semantic view is obtained, wherein each pixel in the expected semantic view contains a probability distribution belonging to each semantic category.

[0094] For pixels at the same coordinate position in the actual semantic view and the expected semantic view, extract the probability distribution of the semantic category to which the pixel belongs in each of the two semantic views, and calculate the cross-entropy value between the two probability distributions.

[0095] Obtain the pixels at all coordinate positions in the actual semantic view and the expected semantic view. Sum the cross-entropy values ​​calculated at each coordinate position and divide by the total number of pixels in the semantic view to obtain the average cross-entropy loss value.

[0096] It should be noted that the actual semantic view is the semantic feature map output by the lightweight semantic segmentation network, while the expected semantic view is the result output by the same lightweight semantic segmentation network after semantic segmentation of the expected appearance image. The two semantic views have the same size and the same semantic category system, and the semantic category probability distribution of each pixel adopts the same encoding method to ensure that the basis for subsequent pixel-by-pixel difference comparison is consistent.

[0097] It should be noted that the calculation process of cross-entropy is as follows: For pixels at the same coordinate position in the actual semantic view and the expected semantic view, obtain the probability value of the pixel belonging to each semantic category in the actual semantic view, and the probability value of the pixel belonging to each semantic category in the expected semantic view; for each semantic category, calculate the logarithm of the ratio of the probability value of the pixel belonging to the category in the expected semantic view to the probability value of the pixel belonging to the category in the actual semantic view, and multiply the logarithm by the probability value in the expected semantic view; sum up the above products corresponding to all semantic categories and take the negative value to obtain the cross-entropy value of the pixel at that coordinate position. The smaller the cross-entropy value, the closer the two probability distributions are, that is, the higher the consistency between the actual semantic view and the expected semantic view at that pixel position.

[0098] It should be noted that the average cross-entropy loss value is calculated as follows: traverse all height coordinates and all width coordinates of the semantic view; for each pixel at each coordinate position, multiply the logarithm of the ratio of the probability value of the pixel belonging to the k-th semantic category in the expected semantic view to the probability value of the pixel belonging to the k-th semantic category in the actual semantic view; sum this product over all semantic categories and take the negative value to obtain the cross-entropy value at that coordinate position; sum the cross-entropy values ​​of all coordinate positions and divide by the product of the height and width of the semantic view to obtain the average cross-entropy loss value.

[0099] It should be noted that the average cross-entropy loss value is used to measure the overall difference between the actual semantic view and the expected semantic view. The smaller the value, the more consistent the actual scene currently collected is with the scene expected to be seen based on the current initial pose estimation, that is, the more accurate the current pose estimation is; conversely, the larger the value, the greater the difference between the actual scene and the expected scene, that is, there is a large error in the current pose estimation, and it is necessary to optimize the pose parameters in reverse to reduce the loss value.

[0100] In the above embodiment, the specific analysis method for setting the difference as the target back-optimization pose parameter to obtain the pose estimate after localization correction is as follows: the average cross-entropy loss value is used as the optimization objective function, the rotation matrix and translation vector in the initial pose estimate of the current frame are used as the pose parameters to be optimized, and the gradient values ​​of each angle component in the rotation matrix and each position component in the translation vector of the average cross-entropy loss value are calculated by the backpropagation algorithm.

[0101] Multiply each gradient value by the preset learning rate to obtain the update amount corresponding to each angle component and each position component. Subtract the corresponding update amount from each angle component and each position component to obtain the updated pose parameters.

[0102] The expected semantic view is regenerated based on the updated pose parameters, and the corresponding average cross-entropy loss value is calculated. If the updated average cross-entropy loss value is greater than the preset convergence threshold, the process is repeated until the average cross-entropy loss value is less than or equal to the convergence threshold. The pose parameters obtained from the last execution are then used as the pose estimate after localization correction.

[0103] It should be noted that the pose parameters to be optimized include a rotation matrix and a translation vector. The rotation matrix contains three angular components: pitch, yaw, and roll. The translation vector contains three positional components: X-axis coordinate, Y-axis coordinate, and Z-axis coordinate, for a total of six degrees of freedom.

[0104] It should be noted that the backpropagation algorithm calculates the partial derivative of the average cross-entropy loss value with respect to the probability distribution of each pixel in the expected semantic view. This partial derivative is then backpropagated through the output layer of the differentiable rendering engine to each component of the extrinsic parameter matrix of the virtual camera. From there, it is backpropagated from the components of the extrinsic parameter matrix to each angular component of the rotation matrix and each positional component of the translation vector, ultimately obtaining the gradient value corresponding to each parameter to be optimized. The positive or negative direction of the gradient value indicates whether the current parameter should be adjusted in the direction of increase or decrease to reduce the loss value.

[0105] It should be noted that the preset learning rate controls the step size of parameter updates in each iteration. A larger learning rate results in a larger parameter adjustment range and a faster convergence speed, but may cause oscillations that may lead to failure to converge. A smaller learning rate results in a smaller parameter adjustment range and a more stable convergence process, but a slower convergence speed. The preset learning rate ranges from one-thousandth to one-tenth, and the specific value should be set according to the actual scenario.

[0106] It should be noted that after each re-execution, the updated pose parameters replace the current pose parameters, and the expected appearance image and expected semantic view are generated again. A new average cross-entropy loss value is calculated, and the new average cross-entropy loss value is compared with a preset convergence threshold. The preset convergence threshold ranges from one percent to one-tenth. When the average cross-entropy loss value is less than or equal to the threshold, it is determined that the actual semantic view and the expected semantic view are sufficiently consistent, the current pose estimation meets the accuracy requirements, execution stops, and the pose parameters obtained from the last execution are used as the pose estimation output after localization correction.

[0107] It should be noted that the entire localization and correction process is as follows: The average cross-entropy loss value of the current frame is used as the optimization objective, and the three angular components of the rotation matrix and the three positional components of the translation vector in the initial pose estimation of the current frame are used as the parameters to be optimized. The gradient of the average cross-entropy loss value with respect to each parameter to be optimized is calculated using the backpropagation algorithm; that is, the direction and magnitude of the average cross-entropy loss value when each parameter changes by a tiny unit. The gradient value of each parameter is multiplied by a preset learning rate to obtain the update amount for each parameter. The original value of each parameter is subtracted from the corresponding update amount to obtain the updated pose parameters. Based on the updated pose parameters, the expected appearance image and expected semantic view are regenerated, and the average cross-entropy loss value is recalculated. It is determined whether the new average cross-entropy loss value is less than or equal to a preset convergence threshold. If the condition is met, execution stops, and the current pose parameters are output as the pose estimation after localization and correction; if the condition is not met, execution continues until the convergence condition is met.

[0108] In the above embodiments, the specific analysis method for deciding whether to execute the current inspection task or generate a navigation waypoint sequence based on the deviation between the map coordinates of the inspection target and the standard observation position and standard observation attitude is as follows: extract pixel regions with the semantic category of the inspection target from the semantic feature map to obtain the inspection target region, and locate the center point image coordinates of the inspection target in the preprocessed image frame based on the inspection target region.

[0109] After obtaining the pose estimate after positioning correction, and combining it with the intrinsic parameter matrix of the airborne camera, the center point image coordinates are transformed to the map coordinate system through inverse projection transformation to obtain the map coordinates of the inspection target.

[0110] Read the standard observation position and standard observation attitude corresponding to the current inspection task from the inspection task list, calculate the position deviation vector between the map coordinates of the inspection target and the standard observation position, and calculate the attitude deviation angle between the orientation of the current pose and the standard observation attitude. Combine the position deviation vector and the attitude deviation angle into a six-degree-of-freedom pose alignment deviation.

[0111] Determine whether the six-degree-of-freedom pose alignment deviation is less than a preset alignment threshold; if so, determine that the UAV has aligned with the inspection target, generate an inspection task execution command, control the UAV to execute the current inspection task and record the inspection data; if not, based on the six-degree-of-freedom pose alignment deviation and combined with the standard observation position, uniformly interpolate to generate a sequence of navigation waypoints on the straight path from the current pose to the standard observation position. Each navigation waypoint contains the target position coordinates and the target orientation angle.

[0112] It should be noted that the specific process of extracting the inspection target region from the semantic feature map is as follows: through the semantic category probability distribution of each pixel in the semantic feature map, the pixel with the largest inspection target category probability value is marked as a candidate target pixel. Connectivity analysis is performed on all candidate target pixels, and the connected component with the largest area is extracted as the inspection target region. The geometric center of the inspection target region is obtained to get the image coordinates of the center point.

[0113] It should be noted that the calculation process of the inverse projection transformation is as follows: the image coordinates of the center point are converted into normalized coordinates in the camera coordinate system using the intrinsic parameter matrix of the airborne camera. Then, combined with the rotation matrix and translation vector in the pose estimation after positioning correction, the normalized coordinates are transformed from the camera coordinate system to the map coordinate system to obtain the map coordinates of the target. The map coordinates of the inspected target are three-dimensional coordinates, including longitude coordinates, latitude coordinates and altitude coordinates.

[0114] It should be noted that the six-degree-of-freedom pose alignment deviation is composed as follows: the position deviation vector contains three components, namely the difference between the map coordinates of the inspected target and the standard observation position in the longitude direction, the difference in the latitude direction, and the difference in the altitude direction; the attitude deviation angle contains three components, namely the deviation angle between the pitch angle of the current pose and the pitch angle of the standard observation pose, the deviation angle between the yaw angle of the current pose and the yaw angle of the standard observation pose, and the deviation angle between the roll angle of the current pose and the roll angle of the standard observation pose; the three components of the position deviation vector and the three components of the attitude deviation angle are combined into a six-dimensional vector to obtain the six-degree-of-freedom pose alignment deviation, which fully describes all the deviation information between the current observation pose and the ideal observation pose.

[0115] It should be noted that the preset alignment threshold is a six-dimensional vector threshold, which corresponds to the upper limit of the allowable deviation of each of the six components of the six-degree-of-freedom pose alignment deviation. When each component of the six-degree-of-freedom pose alignment deviation is less than the corresponding threshold component, it is determined that the UAV has aligned with the inspection target. The navigation waypoint sequence is generated by interpolating the straight path from the current pose to the standard observation position according to the preset waypoint spacing. At each interpolation point, the corresponding position coordinates and orientation angle are calculated, so that the UAV can gradually approach the standard observation position along the navigation waypoint sequence and meet the requirements of the standard observation attitude when reaching the final waypoint.

[0116] In the above embodiments, the specific analysis method for semantic adaptive obstacle avoidance and dynamic waypoint adjustment is as follows: during flight along the navigation waypoint sequence, the current frame image is acquired in real time by the airborne camera to obtain the second-scale semantic feature map of the current frame. Pixel regions with the semantic category of obstacle are extracted from the second-scale semantic feature map to obtain the obstacle region. The spatial range occupied by the obstacle is calculated based on the obstacle region, and it is detected whether the spatial range of the obstacle interferes with the target path of the next waypoint in the navigation waypoint sequence. If no spatial interference occurs, flight continues along the original navigation waypoint sequence.

[0117] If spatial interference occurs, the semantic category of the obstacle is identified from the second-scale semantic feature map, and it is determined whether the semantic category of the obstacle belongs to the dynamic object category or the static obstacle category.

[0118] If the semantic category of the obstacle belongs to the dynamic object category, a hovering waiting command is generated to control the drone to hover in place until the obstacle is detected to have left the interference area, after which it resumes flight along the original navigation waypoint sequence.

[0119] If the semantic category of the obstacle belongs to the static obstacle category, then based on the current UAV pose and the target waypoint position in the navigation waypoint sequence, the A* algorithm is used to perform local path replanning on the obstacle-occupied grid of the second-scale semantic feature map to generate a detour waypoint sequence, which replaces the interference segment in the original navigation waypoint sequence.

[0120] It should be noted that the second-scale semantic feature map is derived from the medium-resolution scale of the three-scale semantic feature maps output by the lightweight semantic segmentation network. The spatial size of the second-scale semantic feature map is two to four times the spatial size of the first-scale semantic feature map. This approach offers high processing speed while preserving sufficient semantic boundary accuracy, meeting the low-latency requirements of real-time obstacle avoidance scenarios. The semantic feature maps at the other two scales are used for global semantic consistency comparison and fine-grained target localization, respectively. All three scales are output simultaneously by the same semantic segmentation network, incurring no additional computational overhead.

[0121] It should be noted that the lightweight semantic segmentation network adopts a multi-scale output structure, simultaneously outputting three semantic feature maps at different resolutions, denoted as the first-scale semantic feature map, the second-scale semantic feature map, and the third-scale semantic feature map. The resolution relationship among the three scales is as follows: the first-scale semantic feature map is the highest resolution scale, with its spatial size being the same as the original size of the preprocessed image frame. It does not undergo downsampling and retains the finest semantic boundary information, making it suitable for tasks requiring fine localization, such as mask extraction of inspection target regions and calculation of pose alignment deviations; the third-scale semantic feature map is the lowest resolution scale, with its spatial size being the first-scale semantic feature map divided by 4 to 8, possessing the largest receptive field and the least computational cost, making it suitable for tasks requiring global semantic context, such as semantic consistency comparison; the second-scale semantic feature map is a medium resolution scale, with its spatial size between the first-scale and third-scale semantic feature maps, being the first-scale semantic feature map divided by 2 to 4, preserving sufficient semantic details while achieving high processing speed.

[0122] It should be noted that the obstacle categories are predefined according to the inspection scenario, including dynamic object categories and static obstacle categories. Among them, the dynamic object category includes pedestrians, vehicles and animals, and the static obstacle category includes buildings, trees, utility poles and temporary construction facilities. The obstacle area is a binary mask, with the obstacle pixel position set to one and the non-obstacle pixel position set to zero. Based on the obstacle area and combined with the depth information or height information in the pose estimation of the current frame, the two-dimensional pixel area is back-projected to the three-dimensional space to calculate the occupancy range of the obstacle in the horizontal and vertical directions, thus obtaining the spatial range.

[0123] It should be noted that the spatial interference detection method is as follows: the target path of the next waypoint in the navigation waypoint sequence is represented as a straight line segment from the current UAV position to the target waypoint position. The minimum distance between the straight line segment and the obstacle space range in three-dimensional space is calculated. If the minimum distance is less than the preset safe distance threshold, spatial interference is determined to have occurred.

[0124] It should be noted that the semantic category of obstacles is identified by counting the number of pixels of each semantic category of obstacles in all pixels within the obstacle area, and taking the semantic category with the most pixels as the semantic category of the obstacle.

[0125] It should be noted that the hovering command keeps the drone in its current position and orientation and continuously collects image frames at a preset frequency to detect the spatial range of obstacles. When the spatial range of the obstacle no longer interferes with the target path after a preset number of consecutive frames, it is determined that the obstacle has left the interference area.

[0126] It should be noted that the local path replanning process of the A* algorithm on the semantic feature map is as follows: using the second-scale semantic feature map as the cost map, the grid cost corresponding to the obstacle pixel position in the obstacle region is set to infinity, and the grid cost of the non-obstacle region is assigned a gradient value according to the distance from the obstacle boundary, with the closer to the obstacle, the higher the value; using the projection grid of the current UAV position in the map coordinate system as the starting grid and the projection grid of the target waypoint position in the map coordinate system as the ending grid, the A* algorithm is used to search for the path from the starting grid to the ending grid with the minimum cost; the searched path grid sequence is converted into the corresponding map coordinate point sequence in sequence to generate the detour waypoint sequence.

[0127] Reference Figure 2 As shown, the present invention provides a system for executing the machine vision-based UAV autonomous inspection, positioning and navigation method, comprising: an image acquisition module: acquiring real-time image frames of the UAV inspection area, extracting features from the image frames, and obtaining a semantic feature map and a geometric feature description set.

[0128] The matching path module: Based on the obtained semantic feature map and geometric feature description set, it performs semantic-geometric joint matching with the pre-stored inspection area map to obtain frame-level pose estimation. The frame-level pose estimation of multiple consecutive frames is concatenated into a pose trajectory sequence. The pose trajectory sequence is matched with the topological path in the pre-stored semantic topology map. The map coordinates of the corresponding topological nodes on the matching path are used as global position constraints, and the frame-level pose estimation is corrected in reverse to obtain the initial pose estimation.

[0129] Inspection target identification module: Based on the standard observation position, standard observation posture and standard appearance model of the inspected target corresponding to the current inspection task in the pre-stored inspection task list, and combined with the initial pose estimation, the expected semantic view is generated. The semantic feature map is used as the actual semantic view. The difference between the actual semantic view and the expected semantic view is calculated. The difference is set as the target back optimization pose parameters to obtain the pose estimate after localization correction.

[0130] The adaptive obstacle avoidance and dynamic waypoint adjustment module locates the map coordinates of the inspection target based on the corrected pose estimation and semantic feature map. According to the deviation between the map coordinates of the inspection target and the standard observation position and standard observation attitude, it decides to execute the current inspection task or generate a navigation waypoint sequence, and performs semantic adaptive obstacle avoidance and dynamic waypoint adjustment.

[0131] It should also be noted that the various threshold settings described in this invention are based on historical data of UAV operation, fault cases, and a large number of experimental test results. At the same time, the thresholds are calibrated by combining the experience of experts in the field and comprehensively considering factors such as the actual operating environment and path performance indicators. This has a solid scientific basis and practical operability, and the technology is relatively mature in the prior art, so it will not be elaborated here.

[0132] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0133] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0134] The above content is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, and all such modifications or additions should fall within the protection scope of the present invention.

Claims

1. A machine vision-based autonomous inspection, positioning, and navigation method for unmanned aerial vehicles (UAVs), characterized in that, include: Step 1, Image Acquisition: Acquire real-time image frames of the UAV inspection area, extract features from the image frames, and obtain semantic feature maps and geometric feature description sets; Step 2, Path Matching: Based on the obtained semantic feature map and geometric feature description set, semantic-geometric joint matching is performed with the pre-stored inspection area map to obtain frame-level pose estimation. The frame-level pose estimations of multiple consecutive frames are concatenated into a pose trajectory sequence. The pose trajectory sequence is matched with the topological path in the pre-stored semantic topology map. The map coordinates of the corresponding topological nodes on the matching path are used as global position constraints, and the frame-level pose estimation is corrected in reverse to obtain the initial pose estimation. Step 3, Inspection Target Identification: Based on the standard observation position, standard observation posture and standard appearance model of the inspected target corresponding to the current inspection task in the pre-stored inspection task list, and combined with the initial pose estimation, the expected semantic view is generated. The semantic feature map is used as the actual semantic view. The difference between the actual semantic view and the expected semantic view is calculated. The difference is set as the target back optimization pose parameters to obtain the pose estimate after localization correction. Step 4, Adaptive obstacle avoidance and dynamic waypoint adjustment: Based on the corrected pose estimation and semantic feature map, locate the map coordinates of the inspection target. According to the deviation between the map coordinates of the inspection target and the standard observation position and standard observation attitude, decide whether to execute the current inspection task or generate a navigation waypoint sequence, and perform semantic adaptive obstacle avoidance and dynamic waypoint adjustment.

2. The machine vision-based autonomous inspection, positioning, and navigation method for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The specific analysis method for extracting features from image frames to obtain semantic feature maps and geometric feature description sets is as follows: The image sequence of the drone inspection area is acquired in real time by an airborne camera. Each frame of the image is denoised and histogram equalized, and then uniformly scaled to a preset size to obtain a preprocessed image frame. The preprocessed image frame is then input into a lightweight semantic segmentation network to extract semantic feature maps. The semantic feature map contains the probability distribution of each pixel belonging to each semantic category. Corner features and line segment features are extracted from the preprocessed image frames, and corner descriptor subsets and line segment descriptor subsets are constructed respectively, and then merged into a geometric feature descriptor set.

3. The machine vision-based autonomous inspection, positioning, and navigation method for unmanned aerial vehicles (UAVs) according to claim 2, characterized in that, The obtained semantic feature map and geometric feature description set are used to perform semantic-geometric joint matching with the pre-stored inspection area map to obtain frame-level pose estimation. The frame-level pose estimations of multiple consecutive frames are then concatenated into a pose trajectory sequence. The specific analysis method is as follows: Read the pre-stored inspection area map, which contains a map semantic feature map and a set of map geometric feature descriptions. Compare the semantic feature map with the map semantic feature map to obtain a semantic similarity score. Select the area with the score exceeding the preset threshold as the semantic matching area. Within the semantic matching region, feature matching is performed between the geometric feature description set and the map geometric feature description set to obtain a candidate matching pair set. Random sampling consistency screening is performed on the candidate matching pair set to remove mismatched pairs, resulting in a refined matching pair set. The rotation matrix and translation vector of the current image frame with respect to the map coordinate system are calculated to obtain the frame-level pose estimation. Obtain frame-level pose estimates for the most recent N consecutive frames of images, concatenate them in chronological order to form a pose trajectory sequence, where N is the preset trajectory window length.

4. The machine vision-based autonomous inspection, positioning, and navigation method for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The process involves sequence matching of the pose trajectory sequence with the topological paths in a pre-stored semantic topology map. The map coordinates of the corresponding topological nodes on the matching path are used as global position constraints, and the frame-level pose estimation is corrected in reverse to obtain the initial pose estimation. The specific analysis method is as follows: Read the pre-stored semantic topology graph, where the semantic topology graph uses semantically significant areas in the inspection area map as topology nodes and the passable paths between semantically significant areas as topology edges. Each topology node is associated with the map coordinates and semantic fingerprint vector of the area. The semantic fingerprint vector is a histogram of the distribution of various semantic categories within a preset radius around the area. Each topology edge is composed of a group of continuous topology nodes in the order of passage, forming a topology path. The pose trajectory sequence is matched with each topological path in the semantic topology graph. The cumulative matching cost between the pose trajectory sequence and each topological path is calculated. The topological path with the minimum cumulative matching cost is selected as the matching path. The map coordinates of the topological nodes at the corresponding positions on the matching path are used as global position constraints. Calculate the position deviation between the global position constraint and the corresponding frame-level pose estimation in the pose trajectory sequence, and use the position deviation as a correction amount to compensate for the frame-level pose estimation of each frame to obtain the initial pose estimation after inverse correction.

5. The machine vision-based autonomous inspection, positioning, and navigation method for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The method for generating the expected semantic view based on the standard observation position, standard observation posture, and standard appearance model of the inspected target corresponding to the current inspection task in the pre-stored inspection task list, combined with the initial pose estimation, is as follows: Obtain a pre-defined list of inspection tasks, and read the standard observation position, standard observation posture, and standard appearance model of the target being inspected corresponding to the current inspection task from the list of inspection tasks. Based on the initial pose estimation, the imaging parameters of the current camera are determined. Combined with the standard observation position and standard observation attitude, the relative spatial relationship between the airborne camera and the inspected target under the current pose is calculated, including the relative position vector and the relative orientation angle. Based on relative spatial relationships and standard appearance models, the standard appearance model is projected onto a two-dimensional imaging plane using a differentiable rendering engine to generate the expected appearance image that should theoretically be captured in the current pose. The expected appearance image is input into a lightweight semantic segmentation network, and pixel-by-pixel semantic annotation is performed on the expected appearance image to obtain the expected semantic view. Each pixel in the expected semantic view contains the corresponding semantic category probability distribution.

6. The machine vision-based autonomous inspection, positioning, and navigation method for unmanned aerial vehicles (UAVs) according to claim 5, characterized in that, The specific analysis method for calculating the difference between the actual semantic view and the expected semantic view is as follows: Based on the obtained semantic feature map, the actual semantic view is used, where each pixel in the actual semantic view contains the probability distribution of each semantic category, and the expected semantic view is obtained, where each pixel in the expected semantic view contains the probability distribution of each semantic category. For pixels at the same coordinate position in the actual semantic view and the expected semantic view, extract the probability distribution of the semantic category to which the pixel belongs in the two semantic views, and calculate the cross-entropy value between the two probability distributions. Obtain the pixels at all coordinate positions in the actual semantic view and the expected semantic view. Sum the cross-entropy values ​​calculated at each coordinate position and divide by the total number of pixels in the semantic view to obtain the average cross-entropy loss value.

7. A machine vision-based autonomous inspection, positioning, and navigation method for unmanned aerial vehicles (UAVs) according to claim 6, characterized in that, The specific analysis method for setting the difference as the target inverse optimization pose parameter to obtain the pose estimate after positioning correction is as follows: The average cross-entropy loss is used as the objective function for optimization. The rotation matrix and translation vector in the initial pose estimation of the current frame are used as the pose parameters to be optimized. The gradient values ​​of each angular component in the rotation matrix and each position component in the translation vector of the average cross-entropy loss are calculated by the backpropagation algorithm. Multiply each gradient value by the preset learning rate to obtain the update amount corresponding to each angle component and each position component. Subtract the corresponding update amount from each angle component and each position component to obtain the updated pose parameters. The expected semantic view is regenerated based on the updated pose parameters, and the corresponding average cross-entropy loss value is calculated. If the updated average cross-entropy loss value is greater than the preset convergence threshold, the process is repeated until the average cross-entropy loss value is less than or equal to the convergence threshold. The pose parameters obtained from the last execution are then used as the pose estimate after localization correction.

8. The machine vision-based autonomous inspection, positioning, and navigation method for unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The method for deciding whether to execute the current inspection task or generate a navigation waypoint sequence based on the deviation between the map coordinates of the inspection target and the standard observation position and standard observation attitude is as follows: Pixel regions with semantic category of inspection target are extracted from the semantic feature map to obtain the inspection target region. Based on the inspection target region, the center point image coordinates of the inspection target are located in the preprocessed image frame. After obtaining the pose estimate after positioning correction, and combining it with the intrinsic parameter matrix of the airborne camera, the center point image coordinates are transformed to the map coordinate system through inverse projection transformation to obtain the map coordinates of the inspection target. Read the standard observation position and standard observation attitude corresponding to the current inspection task from the inspection task list, calculate the position deviation vector between the map coordinates of the inspection target and the standard observation position, and calculate the attitude deviation angle between the orientation of the current pose and the standard observation attitude. Combine the position deviation vector and the attitude deviation angle into a six-degree-of-freedom pose alignment deviation. Determine whether the six-degree-of-freedom pose alignment deviation is less than the preset alignment threshold; If so, it is determined that the drone has aligned with the inspection target, an inspection task execution command is generated, the drone is controlled to execute the current inspection task and record the inspection data; If not, based on the six-degree-of-freedom pose alignment deviation and combined with the standard observation position, a sequence of navigation waypoints is generated by uniform interpolation along the straight path from the current pose to the standard observation position. Each navigation waypoint contains the target position coordinates and the target orientation angle.

9. A machine vision-based autonomous inspection, positioning, and navigation method for unmanned aerial vehicles (UAVs) according to claim 8, characterized in that, The specific analysis method for semantic adaptive obstacle avoidance and dynamic waypoint adjustment is as follows: During flight along the navigation waypoint sequence, the airborne camera acquires the current frame image in real time, obtains the second-scale semantic feature map of the current frame, extracts the pixel region whose semantic category is obstacle from the second-scale semantic feature map, obtains the obstacle region, calculates the spatial range occupied by the obstacle in space based on the obstacle region, and detects whether the spatial range of the obstacle interferes with the target path of the next waypoint in the navigation waypoint sequence. If no spatial interference occurs, continue flying along the original navigation waypoint sequence; If spatial interference occurs, the semantic category of the obstacle is identified from the second-scale semantic feature map, and it is determined whether the semantic category of the obstacle belongs to the dynamic object category or the static obstacle category. If the semantic category of the obstacle belongs to the dynamic object category, a hovering waiting command is generated to control the drone to hover in place until the obstacle is detected to have left the interference area, and then resume flight along the original navigation waypoint sequence. If the semantic category of the obstacle belongs to the static obstacle category, then based on the current UAV pose and the target waypoint position in the navigation waypoint sequence, the A* algorithm is used to perform local path replanning on the obstacle-occupied grid of the second-scale semantic feature map to generate a detour waypoint sequence, which replaces the interference segment in the original navigation waypoint sequence.

10. A system utilizing the machine vision-based unmanned aerial vehicle (UAV) autonomous inspection, positioning, and navigation method according to any one of claims 1-9, characterized in that, include: Image acquisition module: acquires real-time image frames of the UAV inspection area, extracts features from the image frames, and obtains semantic feature maps and geometric feature description sets; Matching Path Module: Based on the obtained semantic feature map and geometric feature description set, semantic-geometric joint matching is performed with the pre-stored inspection area map to obtain frame-level pose estimation. The frame-level pose estimations of multiple consecutive frames are concatenated into a pose trajectory sequence. The pose trajectory sequence is matched with the topological path in the pre-stored semantic topology map. The map coordinates of the corresponding topological nodes on the matching path are used as global position constraints, and the frame-level pose estimation is corrected in reverse to obtain the initial pose estimation. Inspection target recognition module: Based on the standard observation position, standard observation posture and standard appearance model of the inspected target corresponding to the current inspection task in the pre-stored inspection task list, and combined with the initial pose estimation, the expected semantic view is generated. The semantic feature map is used as the actual semantic view. The difference between the actual semantic view and the expected semantic view is calculated. The difference is set as the target back optimization pose parameters to obtain the pose estimate after localization correction. The adaptive obstacle avoidance and dynamic waypoint adjustment module locates the map coordinates of the inspection target based on the corrected pose estimation and semantic feature map. According to the deviation between the map coordinates of the inspection target and the standard observation position and standard observation attitude, it decides to execute the current inspection task or generate a navigation waypoint sequence, and performs semantic adaptive obstacle avoidance and dynamic waypoint adjustment.

Images