An unmanned aerial vehicle visual navigation method based on visual semantic perception

Abstract

The application discloses a kind of unmanned aerial vehicle visual navigation methods based on visual semantic perception, it is related to unmanned aerial vehicle navigation technical field, comprising the following steps: S1, acquisition visual data and pre-processing;S2, the time sequence similar distance of adjacent window fragment is calculated, and the spectral clustering processing is carried out to similarity matrix;S3, with semantic segmentation network as backbone, carry out pixel-level semantic division and carry out correction amount regression, adopt joint loss update network parameter, construct visual navigation model;S4, to current visual sequence is forward inference, to the state recursion of track correction amount is carried out and is observed correction;S5, the track transfer probability between continuous frame is calculated, global optimization is carried out and path likelihood revaluation is carried out;S6, unmanned aerial vehicle navigation control is executed, acquisition execution feedback data and to visual navigation model is incrementally updated.The application can realize unmanned aerial vehicle track correction processing, improve semantic obstacle avoidance accuracy under complex flight scene.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) navigation technology, and in particular to a visual navigation method for UAVs based on visual semantic perception. Background Technology

[0002] With the increasing application of UAVs in inspection, surveying, emergency reconnaissance, and low-altitude autonomous flight scenarios, visual navigation is gradually becoming an important technology for UAV autonomous positioning and trajectory adjustment. Current UAV visual navigation typically uses onboard camera equipment to collect images of the flight environment, and then combines these images with image feature matching, optical flow estimation, target detection, visual odometry, or simultaneous localization and mapping methods to determine the UAV's current position, flight direction, and local obstacles, providing a basis for navigation and control.

[0003] In existing technologies, some solutions incorporate deep learning models to recognize flight images. These models differentiate scene areas such as roads, buildings, vegetation, and obstacles through semantic segmentation or object detection, and then assist the drone in obstacle avoidance or path adjustment based on the recognition results. This type of method, to some extent, improves upon the sensitivity of traditional geometric feature methods to changes in texture, lighting, and viewpoint, enabling the drone to obtain richer environmental perception information.

[0004] However, existing visual navigation methods still have shortcomings. On the one hand, many methods mainly focus on target recognition or obstacle detection in a single frame image, lacking processing of the temporal relationship between adjacent window segments in a continuous visual sequence. When the UAV experiences image jitter, rapid changes in perspective, sudden changes in illumination, or partial occlusion during flight, the visual perception results are prone to abrupt changes, resulting in a lack of stable basis for subsequent trajectory correction.

[0005] On the other hand, the connection between existing deep learning visual recognition results and UAV trajectory correction is not close enough. Semantic segmentation results often remain at the region recognition level and are not further combined with trajectory correction regression, continuous frame state derivation, transition probability optimization, and feedback update processes. This makes it difficult to directly transform visual semantic information into stable and continuous navigation control basis, which easily leads to problems such as trajectory correction lag, accumulation of local misjudgments, and insufficient navigation continuity in complex scenarios.

[0006] Therefore, how to provide a visual navigation method for UAVs based on visual semantic perception is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] One objective of this invention is to propose a visual navigation method for unmanned aerial vehicles (UAVs) based on visual semantic perception. This invention fully utilizes semantic segmentation networks, dynamic time warping, spectral clustering, Kalman filtering, and the Viterbi algorithm. It details the implementation process of UAV visual data preprocessing, training sample construction, visual navigation model training, recursive correction of trajectory correction, target navigation trajectory optimization, and feedback incremental update. It has the advantages of strong semantic perception capability, good trajectory correction continuity, and high navigation stability in complex scenarios.

[0008] A visual navigation method for unmanned aerial vehicles (UAVs) based on visual semantic perception according to an embodiment of the present invention includes the following steps: S1. Collect visual data during the flight of the UAV and preprocess it to form a standard visual sequence; S2. Using the dynamic time warping method, the temporal similarity distance between adjacent window segments in the standard visual sequence is calculated, and the similarity matrix corresponding to the temporal similarity distance is subjected to spectral clustering to generate a training sample set. S3. Based on the training sample set, the semantic segmentation network is used as the backbone of visual semantic perception to perform pixel-level semantic segmentation of visual segments. The trajectory correction amount is regressed based on the semantic segmentation results. The network parameters are updated by the joint loss of semantic segmentation loss and trajectory correction loss to construct a visual navigation model. S4. Based on the visual navigation model, forward reasoning is performed on the current visual sequence. The Kalman filter method is used to recursively calculate the track correction in consecutive frames, and the recursion results are observed and corrected according to the changes in visual semantics. S5. Based on the track correction amount after observation correction, calculate the track transfer probability between consecutive frames, use the Viterbi algorithm for global optimization, and perform path likelihood re-estimation on tracks with semantic change conflicts to determine the target navigation track. S6. Perform UAV navigation control based on the target navigation trajectory, collect execution feedback data, and use a mini-batch gradient descent algorithm to incrementally update the visual navigation model.

[0009] Optionally, the preprocessing specifically includes: Video frames are extracted and sorted according to the acquisition time of visual data, and duplicate frames, missing frames and frames with abnormal exposure are removed to obtain a valid visual frame sequence. Distortion correction is performed on the effective visual frame sequence based on the UAV camera intrinsic parameters, and the pose offset of adjacent frames is calculated by inter-frame feature point matching to perform geometric registration on jittery frames. The scale of the geometrically registered visual frames is adjusted and the field of view is cropped to adjust the frame size, pixel coordinates and channel arrangement to the input format of the semantic segmentation network. The processed visual frames are encapsulated into a sequence according to the acquisition time, and a frame order index is established for adjacent frames to form a standard visual sequence.

[0010] Optionally, S2 specifically includes: S21. The standard visual sequence is continuously truncated according to the preset window length and window step size to obtain multiple window segments arranged in frame order, and the start frame number and end frame number of each window segment are recorded. S22. Using a dynamic time warping method, frame sequence matching is performed on adjacent window segments, the inter-frame difference value is accumulated along the matching path, and the minimum accumulated result is determined as the temporal similarity distance between adjacent window segments. S23. According to the arrangement order of the window segments, determine the values ​​of the matrix elements based on the temporal similarity distance of each adjacent window segment, and construct a similarity matrix based on the correspondence between temporal similarity distance and similarity degree. S24. Perform spectral clustering on the similarity matrix, merge window segments of the same type according to the clustering results, and combine the merged window segments with the corresponding frame order positions to generate a training sample set.

[0011] Optionally, S22 specifically includes: S221. Arrange two adjacent window segments into a first frame sequence and a second frame sequence according to the frame order, respectively. Establish an m×n frame sequence matching grid based on the number of frames in the first frame sequence and the second frame sequence, and set the search bandwidth of dynamic time warping to 10% to 30% of the length of the longer frame sequence. S222. Within the search bandwidth of the frame order matching grid, divide the visual data of the two frames corresponding to the grid nodes into image blocks of 16×16 to 32×32 according to the same block order, calculate the difference in the mean pixel value and the difference in structural similarity between image blocks at the same position, and obtain the local matching cost in a weighted manner. S223. According to the dynamic time warping recursion rule, select the smaller cumulative value from the left, top and top left directions for recursive accumulation, and obtain the cumulative matching cost of each grid node under the starting point constraint, ending point constraint and monotonically progressive constraint. S224. Backtrack the matching path corresponding to the minimum cumulative matching cost, and normalize the minimum cumulative matching cost based on the length of the matching path to determine the temporal similarity distance between adjacent window segments.

[0012] Optionally, S3 specifically includes: S31. Traverse the visual segments in the training sample set, assign semantic category numbers to each pixel within the visual segment, and calculate the coordinate offset frame by frame based on the original track coordinates and corrected track coordinates corresponding to the same visual segment to obtain the track correction training values, where: The original track coordinates and the corrected track coordinates are the track coordinate sequences marked with the visual segment frame order in the training sample set, respectively. The coordinate offset is obtained by subtracting the original track coordinates from the corrected track coordinates. S32. Using a semantic segmentation network as the backbone of visual semantic perception, the visual segments are processed by encoding downsampling and decoding upsampling. The semantic category probability is obtained by using the Softmax classification method at each pixel position, and pixel-level semantic segmentation is completed according to the semantic category probability. S33. Based on the pixel-level semantic segmentation results, the semantic regions traversed by the original track coordinates are processed into region serialization, and a regression mapping method is used to map the region serialization results into predicted track corrections, wherein: The region serialization process involves extracting the semantic category and category probability of the pixels where the original track coordinates are located according to the frame order, and arranging them to form the region serialization result. S34. The semantic segmentation loss is calculated using cross-entropy loss, and the trajectory correction loss is calculated using smooth L1 loss. The semantic segmentation loss and trajectory correction loss are weighted to form a joint loss, and the network parameters are updated through backpropagation to construct a visual navigation model.

[0013] Optionally, S34 specifically includes: S341. Using the semantic category number as a supervision constraint, select the semantic category probability of the corresponding category pixel by pixel in the pixel-level semantic segmentation result, and use the cross-entropy function to measure the semantic category probability with negative logarithm to obtain the semantic segmentation loss. S342. Calculate the residuals between the predicted trajectory correction and the training values ​​of the trajectory correction, and use the smoothing L1 function to segment and constrain the residuals to obtain the trajectory correction loss. S343. Adjust the loss weights according to the gradient change relationship between semantic segmentation loss and trajectory correction loss, and add the weighted semantic segmentation loss and trajectory correction loss to obtain the joint loss; S344. Perform backpropagation based on the joint loss to update the network parameters in the semantic segmentation network that participate in pixel-level semantic segmentation and trajectory correction regression. After the joint loss meets the convergence condition, fix the network parameters and construct the visual navigation model.

[0014] Optionally, S4 specifically includes: S41. Call the visual navigation model and perform forward calculations on the visual frames in the current visual sequence according to the frame order to obtain the pixel-level semantic segmentation results and track corrections for each visual frame. S42. The trajectory correction amount corresponding to each visual frame is used as the observation of Kalman filtering, and the trajectory correction amount and its change amount of the previous frame are used to form a recursive state. The state of the trajectory correction amount of the current frame is predicted according to the state transition matrix. S43. Perform inter-frame registration on the pixel-level semantic segmentation results of adjacent visual frames, calculate the degree of semantic change between adjacent visual frames using semantic intersection-union ratio, and adjust the observation noise covariance of Kalman filter according to the degree of semantic change. S44. Calculate the residual between the current frame track correction and the state prediction result, and use the Mahalanobis distance test to constrain the observation validity of the residual. Calculate the Kalman gain based on the constrained residual, and use the Kalman gain to perform observation correction on the state prediction result to obtain the corrected track correction for the current frame.

[0015] Optionally, S44 specifically includes: S441. Extract the correction component from the current frame track correction to form the current observation vector, and extract the corresponding prediction correction component from the state prediction result to form the prediction observation vector. Perform a difference operation on the current observation vector and the prediction observation vector to obtain the observation residual. S442. The prediction error covariance obtained during the state prediction process is superimposed with the observation noise covariance to obtain the residual covariance. Mahalanobis distance is then applied to the observation residuals based on the residual covariance. A Mahalanobis distance sequence is formed based on the observation residuals that have been corrected in the preceding frames. The median absolute deviation method is used to update the validity threshold, where: When the Mahalanobis distance exceeds the validity threshold, the observation residuals are subjected to amplitude limiting to obtain the valid observation residuals. When the Mahalanobis distance does not exceed the validity threshold, the observation residual is taken as the valid observation residual; S443. Calculate the Kalman gain based on the prediction error covariance and the observation noise covariance, and use the Kalman gain to weight the effective observation residuals to obtain the state correction amount. S444. The state correction amount is superimposed on the state prediction result to obtain the current frame correction state. The current frame correction state is extracted from the current frame correction state, and the prediction error covariance is updated synchronously as the basis for the recursive state of the next frame track correction amount.

[0016] Optionally, S5 specifically includes: S51. Arrange the observed and corrected track corrections into a track correction sequence according to the frame order, and classify each frame track correction into the corresponding track correction state based on the direction difference and amplitude difference between adjacent frame track corrections. S52. Calculate the transition relationship between adjacent track correction states according to frame order, and use the Laplace smoothing method to process the transition relationship probabilistically to obtain the track transition probability between consecutive frames. S53. Using the track correction state as the search node in the Viterbi algorithm, calculate the observation matching probability based on the observed track correction amount and the matching degree between each track correction state, and recursively calculate the cumulative path probability based on the track transition probability, where: The observation matching probability is the matching confidence between the track correction amount after the observation correction in the current frame and each track correction state, which is used to constrain the Viterbi algorithm in the state selection of the current frame. S54. For search nodes with semantic change conflicts, perform path likelihood re-estimation, and backtrack the highest probability path based on the re-estimated cumulative path probability to determine the target navigation track.

[0017] Optionally, S6 specifically includes: S61. Track and execute continuous track points according to the frame order position of the target navigation track, and execute UAV navigation control according to the position changes between adjacent track points; S62. Collect visual feedback data and actual trajectory data after the UAV executes the target navigation trajectory, and calculate the execution deviation between the actual trajectory data and the target navigation trajectory; S63. Organize the visual feedback data, target navigation track and execution deviation into incremental training samples, and use the execution deviation as the basis for updating the training value of track correction. S64. Extract mini-batch samples from the incremental training samples, use the mini-batch gradient descent algorithm to update the parameters of the visual navigation model, and use the updated visual navigation model for subsequent visual navigation.

[0018] The beneficial effects of this invention are: First, this invention performs temporal similarity analysis on continuous visual data of UAVs through preprocessing, dynamic time warping, and spectral clustering to form a training sample set, thereby reducing the impact of image jitter, viewpoint changes, and local occlusion on visual navigation.

[0019] Secondly, this invention uses a semantic segmentation network as the backbone of visual semantic perception, and uses the pixel-level semantic segmentation results for trajectory correction regression, so that visual semantic information can directly participate in UAV trajectory correction and improve semantic understanding and navigation judgment capabilities in complex environments.

[0020] Finally, this invention uses Kalman filtering, track transition probability, and Viterbi algorithm to recursively correct and globally optimize the track correction results of consecutive frames, thereby reducing track jumps caused by single-frame misjudgments and improving the continuity and stability of UAV visual navigation. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a visual navigation method for unmanned aerial vehicles based on visual semantic perception proposed in this invention; Figure 2 This is a flowchart of the Kalman filter observation residual constraint for a visual navigation method for unmanned aerial vehicles based on visual semantic perception proposed in this invention. Figure 3 This is a flowchart illustrating the target navigation trajectory determination process of a UAV visual navigation method based on visual semantic perception proposed in this invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0023] refer to Figures 1-3 A visual navigation method for unmanned aerial vehicles (UAVs) based on visual semantic perception includes the following steps: S1. Collect visual data during the flight of the UAV and preprocess it to form a standard visual sequence; S2. Using the dynamic time warping method, the temporal similarity distance between adjacent window segments in the standard visual sequence is calculated, and the similarity matrix corresponding to the temporal similarity distance is subjected to spectral clustering to generate a training sample set. S3. Based on the training sample set, the semantic segmentation network is used as the backbone of visual semantic perception to perform pixel-level semantic segmentation of visual segments. The trajectory correction amount is regressed based on the semantic segmentation results. The network parameters are updated by the joint loss of semantic segmentation loss and trajectory correction loss to construct a visual navigation model. S4. Based on the visual navigation model, forward reasoning is performed on the current visual sequence. The Kalman filter method is used to recursively calculate the track correction in consecutive frames, and the recursion results are observed and corrected according to the changes in visual semantics. S5. Based on the track correction amount after observation correction, calculate the track transfer probability between consecutive frames, use the Viterbi algorithm for global optimization, and perform path likelihood re-estimation on tracks with semantic change conflicts to determine the target navigation track. S6. Perform UAV navigation control based on the target navigation trajectory, collect execution feedback data, and use a mini-batch gradient descent algorithm to incrementally update the visual navigation model.

[0024] In this embodiment, the preprocessing specifically includes: Video frames are extracted and sorted according to the acquisition time of visual data, and duplicate frames, missing frames and frames with abnormal exposure are removed to obtain a valid visual frame sequence. Distortion correction is performed on the effective visual frame sequence based on the UAV camera intrinsic parameters, and the pose offset of adjacent frames is calculated by inter-frame feature point matching to perform geometric registration on jittery frames. The scale of the geometrically registered visual frames is adjusted and the field of view is cropped to adjust the frame size, pixel coordinates and channel arrangement to the input format of the semantic segmentation network. The processed visual frames are sequentially packaged according to the acquisition time, and a frame order index is established for adjacent frames to form a standard visual sequence. In this embodiment, visual data is acquired at 25 frames per second, and video frames are extracted and sorted according to the acquisition time. When the time interval between two adjacent frames exceeds 80ms, it is marked as a missing frame. When the structural similarity between two adjacent frames is greater than 0.98, the next frame is removed. When the average brightness of a single frame is lower than 20 or higher than 235, the frame is removed, and a valid visual frame sequence is obtained. Distortion correction is performed on the effective visual frame sequence based on the UAV camera intrinsic parameters, and ORB feature point matching and RANSAC methods are used to calculate the pose offset of adjacent frames, and geometric registration is performed on the jittery frames. After geometric registration, the visual frames are scaled and cropped. Each visual image is adjusted to 640×384 pixels, and the pixel values ​​are scaled from 0 to 255 to between 0 and 1. A frame sequence index is established according to the acquisition time to form a standard visual sequence.

[0025] In this embodiment, S2 specifically includes: S21. The standard visual sequence is continuously truncated according to the preset window length and window step size to obtain multiple window segments arranged in frame order, and the start frame number and end frame number of each window segment are recorded. S22. Using a dynamic time warping method, frame sequence matching is performed on adjacent window segments, the inter-frame difference value is accumulated along the matching path, and the minimum accumulated result is determined as the temporal similarity distance between adjacent window segments. S23. According to the arrangement order of the window segments, determine the values ​​of the matrix elements based on the temporal similarity distance of each adjacent window segment, and construct a similarity matrix based on the correspondence between temporal similarity distance and similarity degree. S24. Perform spectral clustering on the similarity matrix, merge window segments of the same type according to the clustering results, and combine the merged window segments with the corresponding frame order positions to generate a training sample set.

[0026] In this embodiment, S22 specifically includes: S221. Arrange two adjacent window segments into a first frame sequence and a second frame sequence according to the frame order, respectively. Establish an m×n frame sequence matching grid based on the number of frames in the first frame sequence and the second frame sequence, and set the search bandwidth of dynamic time warping to 10% to 30% of the length of the longer frame sequence. S222. Within the search bandwidth of the frame order matching grid, divide the visual data of the two frames corresponding to the grid nodes into image blocks of 16×16 to 32×32 according to the same block order, calculate the difference in the mean pixel value and the difference in structural similarity between image blocks at the same position, and obtain the local matching cost in a weighted manner. S223. According to the dynamic time warping recursion rule, select the smaller cumulative value from the left, top and top left directions for recursive accumulation, and obtain the cumulative matching cost of each grid node under the starting point constraint, ending point constraint and monotonically progressive constraint. S224. Backtrack the matching path corresponding to the minimum cumulative matching cost, and normalize the minimum cumulative matching cost based on the length of the matching path to determine the temporal similarity distance between adjacent window segments.

[0027] In this embodiment, S3 specifically includes: S31. Traverse the visual segments in the training sample set, assign semantic category numbers to each pixel within the visual segment, and calculate the coordinate offset frame by frame based on the original track coordinates and corrected track coordinates corresponding to the same visual segment to obtain the track correction training values, where: The original track coordinates and the corrected track coordinates are the track coordinate sequences marked with the visual segment frame order in the training sample set, respectively. The coordinate offset is obtained by subtracting the original track coordinates from the corrected track coordinates. S32. Using a semantic segmentation network as the backbone of visual semantic perception, the visual segments are processed by encoding downsampling and decoding upsampling. The semantic category probability is obtained by using the Softmax classification method at each pixel position, and pixel-level semantic segmentation is completed according to the semantic category probability. S33. Based on the pixel-level semantic segmentation results, the semantic regions traversed by the original track coordinates are processed into region serialization, and a regression mapping method is used to map the region serialization results into predicted track corrections, wherein: The region serialization process involves extracting the semantic category and category probability of the pixels where the original track coordinates are located according to the frame order, and arranging them to form the region serialization result; S34. The semantic segmentation loss is calculated using cross-entropy loss, and the trajectory correction loss is calculated using smooth L1 loss. The semantic segmentation loss and trajectory correction loss are weighted to form a joint loss, and the network parameters are updated through backpropagation to construct a visual navigation model.

[0028] In this embodiment, S34 specifically includes: S341. Using semantic category number as a supervision constraint, select the semantic category probability of the corresponding category pixel by pixel in the pixel-level semantic segmentation result, and use the cross-entropy function to measure the semantic category probability with negative logarithm to obtain the semantic segmentation loss. S342. Calculate the residuals between the predicted trajectory correction and the training values ​​of the trajectory correction, and use the smoothing L1 function to segment and constrain the residuals to obtain the trajectory correction loss. S343. Adjust the loss weights according to the gradient change relationship between semantic segmentation loss and trajectory correction loss, and add the weighted semantic segmentation loss and trajectory correction loss to obtain the joint loss; S344. Perform backpropagation based on the joint loss to update the network parameters in the semantic segmentation network that participate in pixel-level semantic segmentation and trajectory correction regression. After the joint loss meets the convergence condition, fix the network parameters and construct the visual navigation model.

[0029] In this embodiment, S4 specifically includes: S41. Call the visual navigation model and perform forward calculations on the visual frames in the current visual sequence according to the frame order to obtain the pixel-level semantic segmentation results and track corrections for each visual frame. S42. The trajectory correction amount corresponding to each visual frame is used as the observation of Kalman filtering, and the trajectory correction amount and its change amount of the previous frame are used to form a recursive state. The state of the trajectory correction amount of the current frame is predicted according to the state transition matrix. S43. Perform inter-frame registration on the pixel-level semantic segmentation results of adjacent visual frames, calculate the degree of semantic change between adjacent visual frames using semantic intersection-union ratio, and adjust the observation noise covariance of Kalman filter according to the degree of semantic change. S44. Calculate the residual between the current frame track correction and the state prediction result, and use the Mahalanobis distance test to constrain the observation validity of the residual. Calculate the Kalman gain based on the constrained residual, and use the Kalman gain to perform observation correction on the state prediction result to obtain the corrected track correction for the current frame.

[0030] In this embodiment, S44 specifically includes: S441. Extract the correction component from the current frame track correction to form the current observation vector, and extract the corresponding prediction correction component from the state prediction result to form the prediction observation vector. Perform a difference operation on the current observation vector and the prediction observation vector to obtain the observation residual. S442. The prediction error covariance obtained during the state prediction process is superimposed with the observation noise covariance to obtain the residual covariance. Mahalanobis distance is then applied to the observation residuals based on the residual covariance. A Mahalanobis distance sequence is formed based on the observation residuals that have been corrected in the preceding frames. The median absolute deviation method is used to update the validity threshold, where: When the Mahalanobis distance exceeds the validity threshold, the observation residuals are subjected to amplitude limiting to obtain the valid observation residuals. When the Mahalanobis distance does not exceed the validity threshold, the observation residual is taken as the valid observation residual; S443. Calculate the Kalman gain based on the prediction error covariance and the observation noise covariance, and use the Kalman gain to weight the effective observation residuals to obtain the state correction amount. S444. The state correction amount is superimposed on the state prediction result to obtain the current frame correction state. The current frame correction state is extracted from the current frame correction state, and the prediction error covariance is updated synchronously as the basis for the recursive state of the next frame track correction amount.

[0031] In this embodiment, S5 specifically includes: S51. Arrange the observed and corrected track corrections into a track correction sequence according to the frame order, and classify each frame track correction into the corresponding track correction state based on the direction difference and amplitude difference between adjacent frame track corrections. S52. Calculate the transition relationships between adjacent track correction states according to frame order, and use the Laplace smoothing method to probabilistically process the transition relationships to obtain the track transition probabilities between consecutive frames, specifically including: According to the state type, each track correction state in the track correction quantity sequence is numbered to form a track correction state set, and a transition counting matrix is ​​established based on the number of states in the track correction state set. Traverse the sequence of track correction values ​​sequentially along the frame order, read the track correction state of the previous frame and the track correction state of the next frame, form a state transition pair between the two, and accumulate the corresponding count value of the state transition pair in the transition counting matrix. Traverse each row of the transition counting matrix, add a smoothing term to the count values ​​of each subsequent track correction state under the same preceding track correction state, and summarize all the count values ​​after adding the smoothing term to obtain the total number of transitions corresponding to the preceding track correction state. Based on the total transfer amount, the smooth count values ​​of each smoothed count under the same preceding track correction state are row normalized to obtain the track transfer probability of the preceding track correction state to each subsequent track correction state. S53. Using the track correction state as the search node in the Viterbi algorithm, calculate the observation matching probability based on the observed track correction amount and the matching degree between each track correction state, and recursively calculate the cumulative path probability based on the track transition probability, where: The observation matching probability is the confidence level between the track correction amount after observation correction in the current frame and each track correction state, which is used to constrain the Viterbi algorithm's state selection in the current frame. S54. For search nodes with semantic change conflicts, perform path likelihood re-estimation, and backtrack the highest probability path based on the re-estimated cumulative path probability to determine the target navigation track.

[0032] In this embodiment, S6 specifically includes: S61. Track and execute continuous track points according to the frame order position of the target navigation track, and execute UAV navigation control according to the position changes between adjacent track points; S62. Collect visual feedback data and actual trajectory data after the UAV executes the target navigation trajectory, and calculate the execution deviation between the actual trajectory data and the target navigation trajectory; S63. Organize the visual feedback data, target navigation track and execution deviation into incremental training samples, and use the execution deviation as the basis for updating the training value of track correction. S64. Extract mini-batch samples from the incremental training samples, use the mini-batch gradient descent algorithm to update the parameters of the visual navigation model, and use the updated visual navigation model for subsequent visual navigation.

[0033] Example 1: To verify the feasibility of this invention in practice, it was applied to a low-altitude inspection scenario. In this scenario, the UAV autonomously flies along a predetermined flight path over a continuous area. During the flight, there are tree obstructions, repetitive ground textures, interference from building edges, localized shadows, and short-term changes in intense light. Conventional visual navigation methods are prone to misjudging shadows, vegetation boundaries, or reflective areas as passable areas in such scenarios, causing track corrections to jump between consecutive frames. In severe cases, this can lead to deviations from the intended track or untimely avoidance.

[0034] In this scenario, forward visual data during the drone's flight is first acquired, and the video frames are sorted according to the acquisition time. The visual data sampling frequency is 25 frames / second. After distortion correction, geometric registration, scaling, and field-of-view cropping, each frame image is uniformly adjusted to 640×384 pixels, with pixel values ​​scaled from 0 to 255 to between 0 and 1, forming a standard visual sequence. Subsequently, a dynamic temporal warping method is used to calculate the temporal similarity distance between adjacent window segments. The window length is 12 frames, and the window step size is 6 frames. Spectral clustering is then performed on the similarity matrix to form a training sample set.

[0035] During the model training phase, a semantic segmentation network is used as the backbone of visual semantic perception to perform pixel-level semantic segmentation of visual segments, dividing the image area into passable areas, restricted areas, and obstacle areas. Based on the offset relationship between the original track coordinates and the corrected track coordinates, regression training is performed on the track correction amount, and the network parameters are updated using the joint loss of semantic segmentation loss and track correction loss to construct a visual navigation model.

[0036] During actual flight, the visual navigation model performs forward reasoning on the current visual sequence to obtain track corrections in consecutive frames. Then, a Kalman filter is used to recursively calculate the track corrections, and observation corrections are performed based on changes in visual semantics. For semantic conflict occurring in consecutive frames, the track transition probability is further calculated, and a Viterbi algorithm is used for global optimization. After determining the target navigation track, UAV navigation control is executed.

[0037] During the verification process, 36 segments of low-altitude flight data were selected, with each segment ranging from 800m to 1200m, accumulating approximately 186,000 visual frames. When using conventional visual navigation methods, the track correction fluctuated significantly in segments with dense tree shadows and numerous building edges, with an average lateral deviation of 0.74m and a maximum lateral deviation reaching 1.62m. After adopting this invention, the average lateral deviation was reduced to 0.31m, and the maximum lateral deviation was reduced to 0.86m.

[0038] Regarding the stability of continuous frame track correction, the conventional method has 17.8 track correction jumps per thousand frames, while this invention reduces it to 6.3 per thousand frames. For accidental crossings of obstacle semantic regions, the conventional method experiences 14 such instances in 36 flight data segments, while this invention reduces it to 4.

[0039] Table 1 Comparison of Verification Data for Complex Low-Altitude Visual Navigation

[0040] As can be seen from the data in Table 1, the present invention can combine the visual semantic perception results with the trajectory correction process, reduce the impact of single-frame misjudgment on the navigation results, and improve the continuity and stability of UAV visual navigation in complex scenarios.

[0041] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A visual navigation method for unmanned aerial vehicles (UAVs) based on visual semantic perception, characterized in that, Includes the following steps: S1. Collect visual data during the flight of the UAV and preprocess it to form a standard visual sequence; S2. Using the dynamic time warping method, the temporal similarity distance between adjacent window segments in the standard visual sequence is calculated, and the similarity matrix corresponding to the temporal similarity distance is subjected to spectral clustering to generate a training sample set. S3. Based on the training sample set, the semantic segmentation network is used as the backbone of visual semantic perception to perform pixel-level semantic segmentation of visual segments. The trajectory correction amount is regressed based on the semantic segmentation results. The network parameters are updated by the joint loss of semantic segmentation loss and trajectory correction loss to construct a visual navigation model. S4. Based on the visual navigation model, forward reasoning is performed on the current visual sequence. The Kalman filter method is used to recursively calculate the track correction in consecutive frames, and the recursion results are observed and corrected according to the changes in visual semantics. S5. Based on the track correction amount after observation correction, calculate the track transfer probability between consecutive frames, use the Viterbi algorithm for global optimization, and perform path likelihood re-estimation on tracks with semantic change conflicts to determine the target navigation track. S6. Perform UAV navigation control based on the target navigation trajectory, collect execution feedback data, and use a mini-batch gradient descent algorithm to incrementally update the visual navigation model.

2. The UAV visual navigation method based on visual semantic perception according to claim 1, characterized in that, The preprocessing specifically includes: Video frames are extracted and sorted according to the acquisition time of visual data, and duplicate frames, missing frames and frames with abnormal exposure are removed to obtain a valid visual frame sequence. Distortion correction is performed on the effective visual frame sequence based on the UAV camera intrinsic parameters, and the pose offset of adjacent frames is calculated by inter-frame feature point matching to perform geometric registration on jittery frames. The scale of the geometrically registered visual frames is adjusted and the field of view is cropped to adjust the frame size, pixel coordinates and channel arrangement to the input format of the semantic segmentation network. The processed visual frames are encapsulated into a sequence according to the acquisition time, and a frame order index is established for adjacent frames to form a standard visual sequence.

3. The UAV visual navigation method based on visual semantic perception according to claim 1, characterized in that, S2 specifically includes: S21. The standard visual sequence is continuously truncated according to the preset window length and window step size to obtain multiple window segments arranged in frame order, and the start frame number and end frame number of each window segment are recorded. S22. Using a dynamic time warping method, frame sequence matching is performed on adjacent window segments, the inter-frame difference value is accumulated along the matching path, and the minimum accumulated result is determined as the temporal similarity distance between adjacent window segments. S23. According to the arrangement order of the window segments, determine the values ​​of the matrix elements based on the temporal similarity distance of each adjacent window segment, and construct a similarity matrix based on the correspondence between temporal similarity distance and similarity degree. S24. Perform spectral clustering on the similarity matrix, merge window segments of the same type according to the clustering results, and combine the merged window segments with the corresponding frame order positions to generate a training sample set.

4. The UAV visual navigation method based on visual semantic perception according to claim 3, characterized in that, S22 specifically includes: S221. Arrange two adjacent window segments into a first frame sequence and a second frame sequence according to the frame order, respectively. Establish an m×n frame sequence matching grid based on the number of frames in the first frame sequence and the second frame sequence, and set the search bandwidth of dynamic time warping to 10% to 30% of the length of the longer frame sequence. S222. Within the search bandwidth of the frame order matching grid, divide the visual data of the two frames corresponding to the grid nodes into image blocks of 16×16 to 32×32 according to the same block order, calculate the difference in the mean pixel value and the difference in structural similarity between image blocks at the same position, and obtain the local matching cost in a weighted manner. S223. According to the dynamic time warping recursion rule, select the smaller cumulative value from the left, top and top left directions for recursive accumulation, and obtain the cumulative matching cost of each grid node under the starting point constraint, ending point constraint and monotonically progressive constraint. S224. Backtrack the matching path corresponding to the minimum cumulative matching cost, and normalize the minimum cumulative matching cost based on the length of the matching path to determine the temporal similarity distance between adjacent window segments.

5. The UAV visual navigation method based on visual semantic perception according to claim 1, characterized in that, S3 specifically includes: S31. Traverse the visual segments in the training sample set, assign semantic category numbers to each pixel within the visual segment, and calculate the coordinate offset frame by frame based on the original track coordinates and corrected track coordinates corresponding to the same visual segment to obtain the track correction training values, where: The original track coordinates and the corrected track coordinates are the track coordinate sequences marked with the visual segment frame order in the training sample set, respectively. The coordinate offset is obtained by subtracting the original track coordinates from the corrected track coordinates. S32. Using a semantic segmentation network as the backbone of visual semantic perception, the visual segments are processed by encoding downsampling and decoding upsampling. The semantic category probability is obtained by using the Softmax classification method at each pixel position, and pixel-level semantic segmentation is completed according to the semantic category probability. S33. Based on the pixel-level semantic segmentation results, the semantic regions traversed by the original track coordinates are processed into region serialization, and a regression mapping method is used to map the region serialization results into predicted track corrections, wherein: The region serialization process involves extracting the semantic category and category probability of the pixels where the original track coordinates are located according to the frame order, and arranging them to form the region serialization result. S34. The semantic segmentation loss is calculated using cross-entropy loss, and the trajectory correction loss is calculated using smooth L1 loss. The semantic segmentation loss and trajectory correction loss are weighted to form a joint loss, and the network parameters are updated through backpropagation to construct a visual navigation model.

6. The UAV visual navigation method based on visual semantic perception according to claim 5, characterized in that, S34 specifically includes: S341. Using the semantic category number as a supervision constraint, select the semantic category probability of the corresponding category pixel by pixel in the pixel-level semantic segmentation result, and use the cross-entropy function to measure the semantic category probability with negative logarithm to obtain the semantic segmentation loss. S342. Calculate the residuals between the predicted trajectory correction and the training values ​​of the trajectory correction, and use the smoothing L1 function to segment and constrain the residuals to obtain the trajectory correction loss. S343. Adjust the loss weights according to the gradient change relationship between semantic segmentation loss and trajectory correction loss, and add the weighted semantic segmentation loss and trajectory correction loss to obtain the joint loss; S344. Perform backpropagation based on the joint loss to update the network parameters in the semantic segmentation network that participate in pixel-level semantic segmentation and trajectory correction regression. After the joint loss meets the convergence condition, fix the network parameters and construct the visual navigation model.

7. The UAV visual navigation method based on visual semantic perception according to claim 1, characterized in that, S4 specifically includes: S41. Call the visual navigation model and perform forward calculations on the visual frames in the current visual sequence according to the frame order to obtain the pixel-level semantic segmentation results and track corrections for each visual frame. S42. The trajectory correction amount corresponding to each visual frame is used as the observation of Kalman filtering, and the trajectory correction amount and its change amount of the previous frame are used to form a recursive state. The state of the trajectory correction amount of the current frame is predicted according to the state transition matrix. S43. Perform inter-frame registration on the pixel-level semantic segmentation results of adjacent visual frames, calculate the degree of semantic change between adjacent visual frames using semantic intersection-union ratio, and adjust the observation noise covariance of Kalman filter according to the degree of semantic change. S44. Calculate the residual between the current frame track correction and the state prediction result, and use the Mahalanobis distance test to constrain the observation validity of the residual. Calculate the Kalman gain based on the constrained residual, and use the Kalman gain to perform observation correction on the state prediction result to obtain the corrected track correction for the current frame.

8. The UAV visual navigation method based on visual semantic perception according to claim 7, characterized in that, S44 specifically includes: S441. Extract the correction component from the current frame track correction to form the current observation vector, and extract the corresponding prediction correction component from the state prediction result to form the prediction observation vector. Perform a difference operation on the current observation vector and the prediction observation vector to obtain the observation residual. S442. The prediction error covariance obtained during the state prediction process is superimposed with the observation noise covariance to obtain the residual covariance. Mahalanobis distance is then applied to the observation residuals based on the residual covariance. A Mahalanobis distance sequence is formed based on the observation residuals that have been corrected in the preceding frames. The median absolute deviation method is used to update the validity threshold, where: When the Mahalanobis distance exceeds the validity threshold, the observation residuals are subjected to amplitude limiting to obtain the valid observation residuals. When the Mahalanobis distance does not exceed the validity threshold, the observation residual is taken as the valid observation residual; S443. Calculate the Kalman gain based on the prediction error covariance and the observation noise covariance, and use the Kalman gain to weight the effective observation residuals to obtain the state correction amount. S444. The state correction amount is superimposed on the state prediction result to obtain the current frame correction state. The current frame correction state is extracted from the current frame correction state, and the prediction error covariance is updated synchronously as the basis for the recursive state of the next frame track correction amount.

9. The UAV visual navigation method based on visual semantic perception according to claim 1, characterized in that, S5 specifically includes: S51. Arrange the observed and corrected track corrections into a track correction sequence according to the frame order, and classify each frame track correction into the corresponding track correction state based on the direction difference and amplitude difference between adjacent frame track corrections. S52. Calculate the transition relationship between adjacent track correction states according to frame order, and use the Laplace smoothing method to process the transition relationship probabilistically to obtain the track transition probability between consecutive frames. S53. Using the track correction state as the search node in the Viterbi algorithm, calculate the observation matching probability based on the observed track correction amount and the matching degree between each track correction state, and recursively calculate the cumulative path probability based on the track transition probability, where: The observation matching probability is the matching confidence between the track correction amount after the observation correction in the current frame and each track correction state, which is used to constrain the Viterbi algorithm in the state selection of the current frame. S54. For search nodes with semantic change conflicts, perform path likelihood re-estimation, and backtrack the highest probability path based on the re-estimated cumulative path probability to determine the target navigation track.

10. The UAV visual navigation method based on visual semantic perception according to claim 1, characterized in that, S6 specifically includes: S61. Track and execute continuous track points according to the frame order position of the target navigation track, and execute UAV navigation control according to the position changes between adjacent track points; S62. Collect visual feedback data and actual trajectory data after the UAV executes the target navigation trajectory, and calculate the execution deviation between the actual trajectory data and the target navigation trajectory; S63. Organize the visual feedback data, target navigation track and execution deviation into incremental training samples, and use the execution deviation as the basis for updating the training value of track correction. S64. Extract mini-batch samples from the incremental training samples, use the mini-batch gradient descent algorithm to update the parameters of the visual navigation model, and use the updated visual navigation model for subsequent visual navigation.

Images