Unmanned aerial vehicle fusion positioning method based on reverse perspective verification

Abstract

The application relates to the technical field of unmanned aerial vehicle navigation positioning, and discloses an unmanned aerial vehicle fusion positioning method based on reverse visual angle verification. A multi-directional environment image sequence collected by a multi-directional vision sensor and motion state data of an inertial measurement unit are acquired; feature extraction and tracking are performed on the multi-directional environment image sequence, and the degradation quality scores of the visual channels in all directions are calculated; according to the degradation quality scores and a degradation threshold, an optimal standby visual channel is selected, and a positioning main channel is switched; when switching, cross-view angle feature matching is performed on the overlapping observation area to establish the association between the key frame before switching and the current frame after switching, cross-view angle observation constraints are obtained through geometric constraint verification; the motion state data, the visual re-projection error of the current active visual channel and the cross-view angle observation constraints are input into a factor graph optimizer for joint state estimation, and a fusion positioning result is output. The application realizes online evaluation and adaptive switching of the degradation state of the multi-directional visual channel, and improves the continuity of high-precision positioning of the unmanned aerial vehicle under feature degradation.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) navigation and positioning technology, and in particular to a UAV fusion positioning method based on reverse perspective verification. Background Technology

[0002] Currently, consumer and industrial drones are generally equipped with multi-directional visual sensors covering multiple orientations, and use visual-inertial fusion positioning in areas where satellite signals are blocked. Existing visual-inertial positioning solutions (such as VINS-Fusion and ORB-SLAM3) usually only process single-camera input, or use multiple camera channels after offline calibration, and use the feature matching results of fixed visual channels to constrain inertial pose estimation.

[0003] However, in the configuration of multi-directional vision sensors, different vision channels face different feature degradation conditions, such as facing a white wall in the front, facing strong light in the back, and facing a textureless surface in the side, and degradation occurs dynamically during flight. The above scheme relies on preset degradation conditions or manual channel switching, and cannot make online decisions based on actual visual quality. Degradation of a single channel will cause the entire positioning process to fail, and high-quality visual information from other directions will be wasted.

[0004] Therefore, existing technologies cannot perform online assessment and adaptive switching of the degradation state of multi-directional visual channels, resulting in insufficient continuity of high-precision positioning for UAVs in feature degradation scenarios. Summary of the Invention

[0005] This invention provides a UAV fusion localization method and system based on reverse perspective verification to solve the problem that existing technologies cannot perform online evaluation and adaptive switching of the degradation state of multi-directional visual channels, resulting in insufficient continuity of high-precision UAV localization in feature degradation scenarios.

[0006] To achieve the above objectives, the first aspect of the present invention provides a UAV fusion localization method based on reverse perspective verification, comprising: Acquire multi-directional environmental image sequences synchronously collected by the multi-directional vision sensors mounted on the UAV, as well as motion state data synchronously collected by the inertial measurement unit; ORB feature extraction and tracking are performed on the multi-directional environmental image sequence, and degradation quality scores for each directional visual channel are calculated. Based on the degradation quality score and the preset degradation threshold, the optimal backup visual channel is selected, and the positioning main channel is switched to the optimal backup visual channel. When switching visual channels, feature associations between keyframes before and after switching and the current frame after switching are established through cross-view feature matching for overlapping observation areas before and after switching, and geometric constraint verification is performed on the matching results to obtain cross-view observation constraints. The motion state data, the visual reprojection error of the currently active visual channel, and the cross-view observation constraints are uniformly input into the factor graph optimizer for joint state estimation, and the fused localization result of the UAV is output.

[0007] Preferably, the acquisition of multi-directional environmental image sequences synchronously collected by the multi-directional vision sensor mounted on the UAV, and motion state data synchronously collected by the inertial measurement unit, includes: The multi-directional vision sensor includes a forward vision sensor facing the nose of the drone and a rearward vision sensor facing the tail. They simultaneously acquire environmental images of their respective directions and arrange the environmental images according to the acquisition time to obtain a multi-directional environmental image sequence. Based on the exposure time of each image frame in the multi-directional environmental image sequence, angular velocity and linear acceleration are collected by an inertial measurement unit and filtered, and motion state data are composed in timestamp order.

[0008] Preferably, the step of performing ORB feature extraction and tracking on the multi-directional environmental image sequence and calculating the degradation quality score for each directional visual channel includes: ORB features are extracted from the current image frame of each visual channel, and the extracted ORB feature points are tracked with the previous image frame of the same visual channel to obtain the effective tracking feature points of each visual channel. The ratio of the number of effective tracked feature points to the preset number of reference features is used to obtain a tracking number score. Calculate the average disparity and disparity dispersion of the effective tracked feature points between the current image frame and the previous image frame, and obtain the disparity distribution score based on the average disparity and the disparity dispersion. Geometric consistency verification is performed on the effective tracking feature points, and a matching quality score is obtained based on the proportion of the number of verified interior points to the total number of effective tracking feature points. The degradation quality score for each visual channel is obtained by weighted summation of the tracking quantity score, the disparity distribution score, and the matching quality score.

[0009] Preferably, the step of selecting the optimal backup visual channel based on the degradation quality score and a preset degradation threshold, and switching the primary positioning channel to the optimal backup visual channel, includes: When the degradation quality score of the currently active visual channel is lower than the degradation threshold, candidate channels with degradation quality scores higher than the degradation threshold are selected from visual channels in all directions other than the currently active visual channel. When the candidate channel exists, the candidate channel with the highest degradation quality score is determined as the optimal backup visual channel, and the positioning main channel is switched to the optimal backup visual channel. When no candidate channel exists, the currently active visual channel is maintained as the primary localization channel, and the confidence level of the fused localization result is marked as a downgraded state.

[0010] Preferably, during visual channel switching, establishing feature associations between the keyframe before switching and the current frame after switching by cross-view feature matching for the overlapping observation areas before and after switching includes: Based on the installation pose and field of view of the visual channel before and after switching on the UAV body, the overlap range of the field of view of the two visual channels is determined. When the keyframe before the switch and the current frame after the switch have the same view of the same spatial region, and the shared view region is located within the field of view overlap range, it is determined that there is an overlapping observation region before and after the switch. When the pre-switching visual channel is the visual channel corresponding to the forward visual sensor and the post-switching visual channel is the visual channel corresponding to the backward visual sensor, cross-view feature matching is performed on the overlapping observation area between the pre-switching keyframe acquired by the forward visual sensor and the post-switching current frame acquired by the backward visual sensor.

[0011] Preferably, the step of performing geometric constraint verification on the matching results to obtain cross-view observation constraints includes: ORB feature descriptors are extracted from the keyframe before the switch and the current frame after the switch within the overlapping observation area; Calculate the matching distance between two sets of ORB feature descriptors, and retain candidate matching pairs whose matching distance is less than a preset descriptor distance threshold; The candidate matching pairs are verified by epipolar geometry constraints, and mismatched pairs that do not meet the epipolar geometry constraints are eliminated to obtain the verified cross-view matching pairs. Based on the verified cross-view matching pair, the relative pose transformation from the keyframe coordinate system before the switch to the current frame coordinate system after the switch is calculated, and the relative pose transformation is used as the cross-view observation constraint.

[0012] Preferably, the step of jointly estimating the state by inputting the motion state data, the visual reprojection error of the currently active visual channel, and the cross-view observation constraints into a factor graph optimizer, and outputting the fused localization result of the UAV, includes: Between adjacent keyframes of the currently active visual channel, the angular velocity and linear acceleration in the motion state data are pre-integrated to obtain the relative rotation increment, relative translation increment, and relative velocity increment, and the IMU pre-integration factor is constructed based on the relative rotation increment, the relative translation increment, and the relative velocity increment. Based on the observed pixel coordinates of each feature point in the currently active visual channel and the projected pixel coordinates of each feature point predicted based on the state vector to be estimated, the visual reprojection error of the currently active visual channel is calculated, and a visual reprojection factor is constructed based on the visual reprojection error. When switching visual channels, a cross-view constraint factor is constructed based on the cross-view observation constraint; The cross-view constraint factor is used to constrain the relative pose consistency between the keyframe before the switch and the current frame after the switch. The IMU pre-integration factor, the visual reprojection factor, and the cross-view constraint factor are added to the factor graph optimizer for joint state estimation, and the fused localization result is output.

[0013] Preferably, the UAV fusion localization method based on reverse perspective verification further includes: After the positioning main channel is switched to the optimal backup visual channel, the degradation quality score of the original active visual channel before the switch is continuously monitored. When the degradation quality score of the original active visual channel rises above the degradation threshold and the difference between it and the degradation quality score of the optimal backup visual channel is greater than the preset recovery hysteresis threshold, the positioning main channel is switched back to the original active visual channel. The cross-view feature matching is re-executed on the overlapping observation regions before and after the cutback to obtain the cross-view observation constraints after the cutback, and the cross-view observation constraints after the cutback are input into the factor graph optimizer to participate in joint state estimation.

[0014] A second aspect of the present invention provides a UAV fusion positioning system based on reverse perspective verification, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the UAV fusion positioning method based on reverse perspective verification described above.

[0015] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the UAV fusion positioning method based on reverse perspective verification as described in any of the preceding claims.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention performs ORB feature extraction and tracking on the visual channels of multi-directional environmental image sequences, and obtains a degradation quality score by weighted summation of tracking quantity score, disparity distribution score and matching quality score. It objectively quantifies the visual quality of each visual channel from three dimensions: feature quantity, geometric structure and matching reliability, thereby realizing online evaluation of the degradation status of multi-directional visual channels without relying on preset degradation conditions or manual judgment. This solves the problem that existing solutions rely on preset degradation conditions or manual switching and cannot make decisions online.

[0017] (2) The present invention adaptively selects the optimal backup visual channel and switches the main positioning channel based on the degradation quality score and degradation threshold. When the currently active visual channel degrades, the main positioning channel is switched to other visual channels with higher visual quality, so that the degradation of a single channel will no longer cause the entire positioning process to fail, making full use of the high-quality visual information wasted in other directions. When there is no candidate channel that meets the threshold, the current channel is maintained and the confidence of the result is marked as a downgraded state to avoid accidental switching.

[0018] (3) When switching visual channels, this invention establishes feature association between the key frame before switching and the current frame after switching by cross-view feature matching for the overlapping observation areas before and after switching. After verification by epipolar geometry constraints, the cross-view observation constraints in the form of relative pose transformation are obtained. Thus, the forward observation before switching is verified by the backward view and the consistency constraint in the spatial dimension is established at the switching time. Compared with loop detection that only corrects the cumulative drift in the time dimension, this invention solves the problem of spatial consistency of pose before and after switching in real-time switching and suppresses pose jump at the channel switching point.

[0019] (4) This invention integrates the IMU pre-integration factor, visual reprojection factor and cross-view constraint factor into the factor graph optimizer for joint state estimation, realizing the tight coupling and fusion of inertial, visual and cross-view constraints, and maintaining the continuity and robustness of high-precision UAV positioning in the complex flight process where feature degradation and channel switching coexist. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the UAV fusion positioning method based on reverse perspective verification provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, an embodiment of the present invention provides a UAV fusion localization method based on reverse perspective verification, comprising steps S11 to S15: S11: Acquire a sequence of multi-directional environmental images synchronously collected by the multi-directional vision sensor on the UAV, as well as motion state data synchronously collected by the inertial measurement unit. S12, ORB feature extraction and tracking are performed on the multi-directional environmental image sequence, and the degradation quality score of each directional visual channel is calculated. S13, based on the degradation quality score and the preset degradation threshold, select the optimal backup visual channel and switch the positioning main channel to the optimal backup visual channel; S14, When switching visual channels, for the overlapping observation areas before and after the switch, establish the feature association between the key frame before the switch and the current frame after the switch through cross-view feature matching, and verify the matching results with geometric constraints to obtain cross-view observation constraints. S15, the motion state data, the visual reprojection error of the currently active visual channel, and the cross-view observation constraints are uniformly input into the factor graph optimizer for joint state estimation, and the fusion positioning result of the UAV is output.

[0023] In step S11, the multi-directional environmental image sequence synchronously acquired by the multi-directional vision sensor mounted on the UAV, and the motion state data synchronously acquired by the inertial measurement unit, including: The multi-directional vision sensor includes a forward vision sensor facing the nose of the drone and a rearward vision sensor facing the tail. They simultaneously acquire environmental images of their respective directions and arrange the environmental images according to the acquisition time to obtain a multi-directional environmental image sequence. Based on the exposure time of each image frame in the multi-directional environmental image sequence, angular velocity and linear acceleration are collected by an inertial measurement unit and filtered, and motion state data are composed in timestamp order.

[0024] Specifically, the same hardware synchronization trigger signal is sent to both the forward and rearward vision sensors, causing them to acquire environmental images from the camera's front and rear directions respectively at the same exposure time, with a sampling frequency set to 30 frames per second. The environmental images acquired by the forward vision sensor are arranged chronologically to obtain the image frame sequence for the forward vision channel; similarly, the environmental images acquired by the rearward vision sensor are arranged chronologically to obtain the image frame sequence for the rearward vision channel. The image frame sequences from each vision channel together constitute a multi-directional environmental image sequence. It should be noted that the synchronization trigger and unified arrangement ensure a one-to-one temporal correspondence between the image frames from different vision channels, providing a time reference for subsequent cross-view feature matching.

[0025] Regarding the acquisition and alignment of motion state data, the sampling frequency of the inertial measurement unit (IMU) is higher than that of the visual sensor. In this embodiment, the IMU's sampling frequency is 200 times per second. Using the exposure time of each image frame as a reference, the raw angular velocity and raw linear acceleration output by the IMU are linearly interpolated according to the timestamp to obtain angular velocities and linear accelerations aligned with the exposure times of each image frame. The aligned angular velocities and linear accelerations are then low-pass filtered to remove high-frequency noise. The filtered angular velocities and linear accelerations are arranged in time-stamp order to form motion state data. This motion state data is used for subsequent pre-integration operations between adjacent keyframes and participates in the joint state estimation of the factor graph. For example, a second-order Butterworth low-pass filter with a cutoff frequency of 50Hz is used. In step S12, ORB feature extraction and tracking are performed on the multi-directional environmental image sequence, and degradation quality scores for each directional visual channel are calculated, including: ORB features are extracted from the current image frame of each visual channel, and the extracted ORB feature points are tracked with the previous image frame of the same visual channel to obtain the effective tracking feature points of each visual channel. The ratio of the number of effective tracked feature points to the preset number of reference features is used to obtain a tracking number score. Calculate the average disparity and disparity dispersion of the effective tracked feature points between the current image frame and the previous image frame, and obtain the disparity distribution score based on the average disparity and the disparity dispersion. Geometric consistency verification is performed on the effective tracking feature points, and a matching quality score is obtained based on the proportion of the number of verified interior points to the total number of effective tracking feature points. The degradation quality score for each visual channel is obtained by weighted summation of the tracking quantity score, the disparity distribution score, and the matching quality score.

[0026] It should be noted that this embodiment uses ORB features as the implementation method for feature extraction and tracking; in other embodiments, SIFT features, SURF features, or FAST corner points combined with BRIEF descriptors are also used to achieve feature extraction of environmental images and feature tracking of adjacent frames, so as to obtain effective tracking feature points and achieve the same feature association effect.

[0027] Specifically, a multi-layer image pyramid is constructed for the current image frame. FAST corner points are detected on each layer, and the principal direction of each corner point is calculated. A rotation-invariant BRIEF binary descriptor is generated centered on the corner point to obtain the ORB feature points of the current image frame. The ORB feature points of the current image frame are then matched with the ORB feature points of the previous image frame in the same visual channel according to the descriptor. Feature points that successfully match and pass the bidirectional consistency check between the preceding and following frames are identified as valid tracking feature points for that visual channel. For example, the forward visual channel extracts 500 ORB feature points in the current image frame, and 320 of these are valid tracking feature points that match the previous image frame and pass the bidirectional consistency check.

[0028] When calculating the tracking quantity score, the number of valid tracking feature points is counted, and this number is compared with the preset number of reference features. The upper limit of the ratio is then truncated to 1 to obtain the tracking quantity score, as shown in the following formula:

[0029] In the formula, To track quantity scoring, a dimensionless value ranging from 0 to 1 is used. In this embodiment, the forward visual channel... Example value is 1; To effectively track the number of feature points, the value is dimensionless and ranges from integers greater than or equal to 0. In this embodiment, the example value is 320. The preset number of reference features is dimensionless and ranges from integers greater than 0; in this embodiment, the example value is 150. The number of reference features... The determination is based on the following: For each frame of image frames acquired offline by the target UAV model and its visual sensor in urban canyons, indoor environments, and satellite signal obstruction scenarios, the number of effective tracking feature points when the visual inertial positioning accuracy meets the requirements is statistically analyzed. This sample is then sorted in ascending order, and the 20th percentile is taken as the reference feature count. In this embodiment, the value is set to 150, so that the tracking number score reflects the sufficiency of the current number of features relative to the number of features required for reliable localization. For example, when the visual channel in a certain direction faces a white wall or a textureless surface, the number of effectively tracked feature points... When the score is 45, the number of tracks is scored. It is 0.30.

[0030] When calculating the disparity distribution score, for each effective tracked feature point, the magnitude of its pixel coordinate displacement between the current image frame and the previous image frame is calculated as the disparity of that effective tracked feature point. The arithmetic mean of the disparities of all effective tracked feature points is calculated to obtain the average disparity, and the standard deviation of the disparities of all effective tracked feature points is calculated to obtain the disparity dispersion. The disparity distribution score is obtained based on the average disparity and the disparity dispersion, as shown in the following formula:

[0031] In the formula, The disparity distribution score is dimensionless and ranges from 0 to 1. In this example, the value is 0.587. The average disparity is expressed in pixels and ranges from 0 to 8.5. The preset reference parallax is in pixels and its value ranges from 0 to 5. This represents the disparity dispersion, measured in pixels, with a value range greater than or equal to 0. In this example, the value is 3.2. This is a preset reference disparity, measured in pixels, with a value greater than 0; in this embodiment, the value is 6. The reference disparity... With reference dispersion The determination is based on the following: For the image frames acquired offline, statistical samples of the average disparity and disparity dispersion when the positioning accuracy meets the requirements are used, and the median of the average disparity samples is taken as the reference disparity. Take the median of the disparity dispersion samples as the reference dispersion. A low average disparity indicates a distant scene, flat texture, or weak geometric constraints. A high disparity dispersion indicates that the disparity is caused by a small number of dynamic objects or mistracking, and is unevenly distributed. The disparity distribution score varies depending on the scenario. All decreased accordingly.

[0032] When calculating the matching quality score, the matching pairs formed between two adjacent frames of effective tracked feature points are validated using epipolar geometric constraints as the geometric consistency model: samples are randomly drawn from the matching pairs to estimate the fundamental matrix, and the number of matching pairs that satisfy the epipolar geometric constraints is counted as inliers. After a preset number of iterations of 200, the model with the largest number of inliers is retained, and matching pairs that deviate from this model are judged as outliers and discarded. The proportion of inliers that pass the geometric consistency verification to the total number of effective tracked feature points is used as the matching quality score, as shown in the following formula:

[0033] In the formula, To match the quality score, it is dimensionless and ranges from 0 to 1. In this example, the value is 0.90. The number of interior points that pass the geometric consistency verification is dimensionless and takes the value of an integer greater than or equal to 0. In this example, the value is 288. To effectively track the total number of feature points, a dimensionless value is used, with values ​​ranging from integers greater than 0. In this embodiment, the example value is 320. When the visual channel is affected by strong light, motion blur, or interference from numerous moving objects, the number of interior points... Reduce, match quality score The corresponding reduction.

[0034] When calculating the degradation quality score, the tracking quantity score, disparity distribution score, and matching quality score are weighted and summed according to their respective weights to obtain the degradation quality score for each visual channel in each direction, as shown in the following formula:

[0035] In the formula, The degradation quality score is dimensionless and ranges from 0 to 1. In this embodiment, the forward visual channel... Example value: 0.42, rearward visual channel Example value is 0.86; , , These are the tracking quantity score, disparity distribution score, and matching quality score, respectively. They are dimensionless and range from 0 to 1. , , These are the weights for the tracking quantity score, disparity distribution score, and matching quality score, respectively. They are dimensionless, ranging from 0 to 1, and the sum of the three is 1. In this embodiment, the example values ​​are 0.4, 0.3, and 0.3, respectively. , , The determination is based on the following: For the image frames acquired offline and labeled with positioning errors, a linear regression was performed with tracking quantity score, disparity distribution score, and matching quality score as independent variables and positioning error as the dependent variable. The absolute values ​​of the regression coefficients of each independent variable were normalized and used as the corresponding weights, so that the score items that contribute more to the positioning error received higher weights. The higher the degradation quality score, the better the current visual quality of the visual channel in that direction, and the more suitable it is as the main positioning channel. It should be noted that the example values ​​s1=1, s2=0.587, and s3=0.90 of the above score items are examples of the forward visual channel in a normal scene, while the degradation quality score Q=0.42 of the forward visual channel is an example value of the forward visual channel in a degradation scene, corresponding to different image frames. For example, in a degradation scenario, the forward visual channel has the following characteristics: the number of effective tracked feature points N is 68, the number of reference features N0 is 150, and the tracking number score s1 = 0.45; the average disparity d is 3.5 pixels, the reference disparity d0 is 5 pixels, the disparity dispersion is 6.2 pixels, and the reference dispersion is 6 pixels, resulting in a disparity distribution score s2 = 0.25; the number of inliers M is 110, the total number of effective tracked feature points N is 200, and the matching quality score s3 = 0.55; the degradation quality score Q = 0.4 * 0.45 + 0.3 * 0.25 + 0.3 * 0.55 = 0.42.

[0036] In step S13, based on the degradation quality score and a preset degradation threshold, an optimal backup visual channel is selected, and the primary positioning channel is switched to the optimal backup visual channel, including: When the degradation quality score of the currently active visual channel is lower than the degradation threshold, candidate channels with degradation quality scores higher than the degradation threshold are selected from visual channels in all directions other than the currently active visual channel. When the candidate channel exists, the candidate channel with the highest degradation quality score is determined as the optimal backup visual channel, and the positioning main channel is switched to the optimal backup visual channel. When no candidate channel exists, the currently active visual channel is maintained as the primary localization channel, and the confidence level of the fused localization result is marked as a downgraded state.

[0037] Among them, the positioning master channel is the visual channel currently used to provide visual reprojection constraints to the factor graph optimizer, and the visual channel currently used as the positioning master channel is called the currently active visual channel.

[0038] Specifically, the degradation quality score of the currently active visual channel is compared with a preset degradation threshold. When the degradation quality score of the currently active visual channel is lower than the degradation threshold, it is determined that the currently active visual channel has degraded. From the visual channels in all directions other than the currently active visual channel, visual channels with degradation quality scores higher than the degradation threshold are selected as candidate channels. For example, the degradation threshold is set to 0.5. When the currently active visual channel is the forward visual channel and its degradation quality score is 0.42, which is lower than the degradation threshold of 0.5, it is determined that the forward visual channel has degraded. The degradation quality score of the backward visual channel is 0.86, which is higher than the degradation threshold of 0.5, and it is selected as a candidate channel. The degradation threshold is determined as follows: each of the offline acquired image frames is labeled with a visual degradation label. Using the degradation quality score as the criterion, the threshold is tried one by one in the range of 0 to 1 with a step size of 0.05, and the classification accuracy of normal frames and degraded frames is counted. The score value corresponding to the highest classification accuracy is taken as the degradation threshold. In this embodiment, the value is 0.5.

[0039] When candidate channels exist, the degradation quality scores of each candidate channel are ranked, and the candidate channel with the highest degradation quality score is selected as the optimal backup visual channel. The primary localization channel is then switched from the currently active visual channel to the optimal backup visual channel. For example, when only the backward visual channel is a candidate channel, the backward visual channel is determined as the optimal backup visual channel, and the primary localization channel is switched from the forward visual channel to the backward visual channel.

[0040] When the degradation quality scores of all visual channels other than the currently active visual channel are not higher than the degradation threshold and there are no candidate channels, the currently active visual channel is maintained as the primary positioning channel, and the confidence level of the fused positioning result is marked as a degraded state. The degraded state is used to indicate to the upper-level navigation or flight control module that there is currently no visual channel that meets the quality requirements and the confidence level of the positioning result has decreased, so that the upper-level module can implement strategies such as deceleration, hovering, or increasing the inertial calculation weight to avoid erroneous switching when there are no reliable candidate channels.

[0041] In one implementation, the UAV fusion localization method based on reverse perspective verification further includes monitoring and switching back to the original active visual channel, including: After the positioning main channel is switched to the optimal backup visual channel, the degradation quality score of the original active visual channel before the switch is continuously monitored. When the degradation quality score of the original active visual channel rises above the degradation threshold and the difference between it and the degradation quality score of the optimal backup visual channel is greater than the preset recovery hysteresis threshold, the positioning main channel is switched back to the original active visual channel. The cross-view feature matching is re-executed on the overlapping observation regions before and after the cutback to obtain the cross-view observation constraints after the cutback, and the cross-view observation constraints after the cutback are input into the factor graph optimizer to participate in joint state estimation.

[0042] Specifically, after switching the primary positioning channel from the original active visual channel to the optimal backup visual channel, the degradation quality score of the original active visual channel is calculated and monitored frame by frame according to step S12 to determine whether the visual quality of the original active visual channel has recovered. Here, the original active visual channel is the visual channel that served as the primary positioning channel before the switch, and is distinguished from the optimal backup visual channel that serves as the primary positioning channel after the switch.

[0043] Regarding the timing of the switchback, when the degradation quality score of the original active visual channel rises above the degradation threshold, and the difference between the degradation quality score of the original active visual channel and the degradation quality score of the current optimal backup visual channel used as the primary positioning channel is greater than a preset recovery hysteresis threshold, the primary positioning channel is switched back to the original active visual channel. For example, the recovery hysteresis threshold is set to 0.15. When the degradation quality score of the original forward visual channel rises to 0.83, the degradation quality score of the current backward visual channel is 0.66, the difference between the two is 0.17, which is greater than the recovery hysteresis threshold of 0.15, the primary positioning channel is switched back to the forward visual channel. The recovery hysteresis threshold is determined by calculating the standard deviation of the score difference between adjacent frames in the time series of degradation quality scores for the same visual channel during a stable flight segment. Twice of this standard deviation is taken as the recovery hysteresis threshold. In this embodiment, the value is 0.15, so that the switchback only occurs when the score of the original active visual channel is significantly better than that of the current main channel, avoiding frequent switching caused by repeated jittering near the degradation threshold when the scores of the two visual channels are close.

[0044] When switching the main positioning channel back to the original active visual channel, the cross-view feature matching and geometric constraint verification are re-executed for the overlapping observation areas before and after the switch back according to the process in step S14, so as to obtain the cross-view observation constraints after the switch back. The cross-view observation constraints after the switch back are constructed into cross-view constraint factors in the same way as the cross-view constraint factors in step S15 and input into the factor graph optimizer to participate in joint state estimation, so that the positioning results at the time of the switch back maintain spatial consistency.

[0045] In step S14, during visual channel switching, for the overlapping observation areas before and after the switching, feature associations between the keyframe before the switching and the current frame after the switching are established through cross-view feature matching, and the matching results are geometrically constrained to obtain cross-view observation constraints, including: Based on the installation pose and field of view of the visual channel before and after switching on the UAV body, the overlap range of the field of view of the two visual channels is determined. When the keyframe before the switch and the current frame after the switch have the same view of the same spatial region, and the shared view region is located within the field of view overlap range, it is determined that there is an overlapping observation region before and after the switch. When the pre-switching visual channel is the visual channel corresponding to the forward visual sensor and the post-switching visual channel is the visual channel corresponding to the backward visual sensor, cross-view feature matching is performed on the overlapping observation area between the pre-switching keyframe acquired by the forward visual sensor and the post-switching current frame acquired by the backward visual sensor.

[0046] Among them, the last image frame selected as a key frame in the pre-switching visual channel before the switching occurs is called the pre-switching key frame, and the image frame acquired by the post-switching visual channel at the switching moment is called the post-switching current frame. This step uses the post-switching visual channel to perform reverse perspective verification of the observations of the pre-switching visual channel, and establishes spatial consistency constraints at the switching moment.

[0047] Specifically, the mounting pose and field of view of the visual channel before and after the switch in the UAV's body coordinate system are read. The mounting pose includes the mounting position and the orientation of the optical axis. Combined with the current pose of the UAV, the visual cones of the two visual channels are projected onto the same spatial coordinate system, and the area where the two visual cones intersect in space is taken as the overlapping range of the fields of view of the two visual channels. For example, when the UAV performs hovering adjustments or turning maneuvers in a narrow passage, the visual cones of the forward visual sensor and the rearward visual sensor form an overlapping range of fields of view on the side of the UAV or in the space that has already been flown.

[0048] When determining the overlapping observation region, the system uses the estimated poses of the keyframe before the switch and the current frame after the switch, along with the 3D coordinates of the observed feature points, to determine whether they observe the same spatial region. If both observe the same spatial region and this observed region is within the aforementioned defined field-of-view overlap range, an overlapping observation region is determined to exist before and after the switch, and this overlapping observation region is used as the scope of cross-view feature matching. If no observed region exists within the field-of-view overlap range, the switch is deemed not to meet the conditions for establishing cross-view observation constraints, and no cross-view constraint factor is constructed. The joint state estimation is maintained by the IMU pre-integration factor and the visual reprojection factor of the channel after the switch. It should be noted that the existence determination of the overlapping observation region ensures that cross-view feature matching is only performed when there is indeed a common observation before and after the switch, avoiding mismatches when there is no common observation.

[0049] When performing cross-view feature matching, when the main positioning channel switches from the forward vision channel to the rear vision channel, the key frame before the switch is the image frame acquired by the forward vision sensor, and the current frame after the switch is the image frame acquired by the rear vision sensor. For the overlapping observation area of ​​the two, cross-view feature matching is performed between the key frame before the switch acquired by the forward vision sensor and the current frame after the switch acquired by the rear vision sensor. The rear vision from the tail direction is used to re-observe and match the already flown space area observed by the forward vision from the nose direction from the reverse perspective.

[0050] In one implementation, the step of performing geometric constraint verification on the matching results to obtain cross-view observation constraints includes: ORB feature descriptors are extracted from the keyframe before the switch and the current frame after the switch within the overlapping observation area; Calculate the matching distance between two sets of ORB feature descriptors, and retain candidate matching pairs whose matching distance is less than a preset descriptor distance threshold; The candidate matching pairs are verified by epipolar geometry constraints, and mismatched pairs that do not meet the epipolar geometry constraints are eliminated to obtain the verified cross-view matching pairs. Based on the verified cross-view matching pair, the relative pose transformation from the keyframe coordinate system before the switch to the current frame coordinate system after the switch is calculated, and the relative pose transformation is used as the cross-view observation constraint.

[0051] Specifically, within the overlapping observation area, ORB feature points are detected and corresponding ORB feature descriptors are calculated for the keyframe before the switch, and ORB feature points are detected and corresponding ORB feature descriptors are calculated for the current frame after the switch. The ORB feature descriptor is a 256-bit binary string, which quantizes the local texture of the feature point by comparing the grayscale relationship of pixels in the neighborhood of the feature point.

[0052] When selecting candidate matching pairs, for each ORB feature descriptor in the keyframe before the switch, the Hamming distance between it and each ORB feature descriptor in the current frame after the switch is calculated as the matching distance. The one with the smallest matching distance is selected as the candidate counterpart for that descriptor. When the matching distance of the candidate counterpart is less than a preset descriptor distance threshold, it is retained as a candidate matching pair; when the matching distance of the candidate counterpart is not less than the preset descriptor distance threshold, it is discarded. For example, the descriptor distance threshold is set to 50. When the Hamming distance of a candidate counterpart is 35, which is less than the descriptor distance threshold of 50, it is retained as a candidate matching pair. The descriptor distance threshold is determined as follows: the Hamming distance distribution of the 256-bit descriptors of the correctly labeled and incorrectly labeled offline pairs is statistically analyzed, and the median of the 95th percentile of the Hamming distance of the correctly labeled pair and the 5th percentile of the Hamming distance of the incorrectly labeled pair is taken as the descriptor distance threshold. In this embodiment, the value is 50. It should be noted that the forward vision sensor and the backward vision sensor have opposite observation perspectives on the same spatial region. Using rotation-invariant ORB descriptors and relaxing the descriptor distance threshold is beneficial for establishing cross-view feature associations under large viewpoint differences.

[0053] In the epipolar geometry constraint verification, the process is as follows: Construction: Epipolar geometry equations are constructed from the corresponding pixels of the keyframe before the switch and the current frame after the switch in the candidate matching pairs. Each pair of matching points provides one constraint equation. Eight pairs of matching points are randomly selected to construct a system of eight equations. Solution: The essential matrix between the keyframe before the switch and the current frame after the switch is solved using singular value decomposition. Substitution: The essential matrix is ​​substituted into the epipolar geometry constraints, and each candidate matching pair is verified to ensure that the matching point is located in the neighborhood of the corresponding epipolar line. Candidate matching pairs that meet the condition are counted as inliers, and those that do not meet the condition are judged as mismatched pairs. Output: After a preset number of iterations of 200, the essential matrix with the largest number of inliers is retained, and its corresponding inliers are used as the verified cross-view matching pairs. For example, after removing 25 mismatched pairs from 120 candidate matching pairs through epipolar geometry constraint verification, 95 verified cross-view matching pairs are obtained.

[0054] When calculating the relative pose transformation, the following steps are taken: Construction: Construct the essential matrix using the validated cross-view matching pairs and camera intrinsic parameters; Solution: Perform singular value decomposition on the essential matrix to obtain the relative rotation and relative translation from the keyframe coordinate system before the switch to the current frame coordinate system after the switch, which together constitute the relative pose transformation; Substitution: Apply the relative pose transformation to the pose of the keyframe before the switch to obtain the pose of the current frame after the switch predicted by cross-view observations; Output: Use this relative pose transformation as a cross-view observation constraint to characterize the relative pose relationship that should be satisfied between the keyframe before the switch and the current frame after the switch, which is used to constrain the spatial consistency of the pose before and after the switch in subsequent factor graph joint state estimation.

[0055] In step S15, the motion state data, the visual reprojection error of the currently active visual channel, and the cross-view observation constraint are input into a unified factor graph optimizer for joint state estimation, and the fused localization result of the UAV is output, including: Between adjacent keyframes of the currently active visual channel, the angular velocity and linear acceleration in the motion state data are pre-integrated to obtain the relative rotation increment, relative translation increment, and relative velocity increment, and the IMU pre-integration factor is constructed based on the relative rotation increment, the relative translation increment, and the relative velocity increment. Based on the observed pixel coordinates of each feature point in the currently active visual channel and the projected pixel coordinates of each feature point predicted based on the state vector to be estimated, the visual reprojection error of the currently active visual channel is calculated, and a visual reprojection factor is constructed based on the visual reprojection error. When switching visual channels, a cross-view constraint factor is constructed based on the cross-view observation constraint; The cross-view constraint factor is used to constrain the relative pose consistency between the keyframe before the switch and the current frame after the switch. The IMU pre-integration factor, the visual reprojection factor, and the cross-view constraint factor are added to the factor graph optimizer for joint state estimation, and the fused localization result is output.

[0056] The factor graph includes state nodes representing the state to be estimated of the UAV at each key frame and factor nodes connecting the state nodes. The state vector to be estimated includes the UAV's position, velocity, attitude, and the zero bias of the inertial measurement unit.

[0057] Specifically, within the time interval corresponding to two adjacent keyframes of the currently activated visual channel, the angular velocity in the motion state data is integrated over time to obtain the relative rotation increment; the linear acceleration, after deducting the gravity component, is integrated over time once to obtain the relative velocity increment, and integrated over time twice to obtain the relative translation increment; the relative rotation increment, relative translation increment, and relative velocity increment are used as pre-integration observations between the states of two adjacent keyframes to construct the IMU pre-integration factor. The IMU pre-integration factor constrains the position, velocity, and attitude states of two adjacent keyframes to satisfy the relative motion relationship deduced from the inertial measurement data.

[0058] When constructing the visual reprojection factor, for each feature point observed by the currently active visual channel, its actual pixel coordinates extracted in the image frame are used as the observed pixel coordinates. Based on the UAV pose in the state vector to be estimated and the 3D coordinates of the feature point, combined with the intrinsic parameters of the visual sensor corresponding to the currently active visual channel and the extrinsic parameters relative to the aircraft, the 3D coordinates of the feature point are projected onto the image plane to obtain the predicted projected pixel coordinates. The difference between the observed pixel coordinates and the predicted projected pixel coordinates is used as the visual reprojection error of the feature point, as shown in the following formula:

[0059] In the formula, , is the visual reprojection error of the j-th feature point in the i-th keyframe, in pixels, and its value ranges from greater than or equal to 0. In this embodiment, the converged example value is 0.6. The pixel coordinates of the j-th feature point in the i-th keyframe are observed in pixels, and the value range is the pixel coordinates in the image plane. In this embodiment, the example values ​​are 320.5 for the horizontal coordinate and 240.2 for the vertical coordinate. This is a projection function that maps 3D coordinates to pixel coordinates. It is a dimensionless mapping operator and does not take into account the range or example values. This is the intrinsic parameter matrix of the visual sensor corresponding to the currently activated visual channel. It is dimensionless and its value range is an upper triangular matrix composed of focal length and principal point coordinates. In this embodiment, the focal length is 460 pixels, the horizontal coordinate of the principal point is 320, and the vertical coordinate of the principal point is 240. Let represent the UAV pose corresponding to the i-th keyframe, where the translation component is in meters and the rotation component is dimensionless, with values ​​ranging from real numbers of the corresponding physical quantities. In this embodiment, the translation components of a certain keyframe are 1507 meters, 2003 meters, and 498 meters. Let be the three-dimensional coordinates of the j-th feature point, in meters, with values ​​ranging from real numbers. Assuming the drone's head is facing the +X axis, the example value in this embodiment is (1509.0, 2004.0, 513.0), representing a feature point approximately 2 meters in front of the drone, 1 meter to the left, and 15 meters above it. In other words, the example value is a local coordinate of the same order of magnitude as the drone's pose.

[0060] For each feature point of the currently active visual channel, the visual reprojection error is calculated and a visual reprojection factor is constructed accordingly. The visual reprojection factor constrains the state vector to be estimated so that the projected pixel coordinates of each feature point are close to their observed pixel coordinates.

[0061] Regarding the construction of cross-view constraint factors, when a visual channel switch occurs and cross-view observation constraints are obtained in step S14, the relative pose between the state node corresponding to the keyframe before the switch and the state node corresponding to the current frame after the switch is constrained by the aforementioned relative pose transformation, thereby constructing cross-view constraint factors. These cross-view constraint factors connect the state nodes corresponding to the keyframe before the switch and the current frame after the switch, ensuring that the relative pose relationship between their estimated poses is consistent with the cross-view observation constraints. The reverse view verification results of the visual channel after the switch on the observation area before the switch are introduced into the joint optimization to suppress pose jumps and drifts at the channel switch point. During periods when no switch occurs or the current switch does not meet the condition for the existence of overlapping observation areas, cross-view constraint factors are not added to the factor graph.

[0062] The residual r3 of the cross-view constraint factor is defined as the six-degree-of-freedom error between the relative pose transformation between the keyframe pose Ti before the switch and the current frame pose Tj after the switch, and the relative pose transformation Tobs given by the cross-view observation constraint obtained in step S14. It is expressed in the tangent space of the special Euclidean group SE(3). The residual term r3 is equal to the product of the inverse of Tobs, the inverse of Ti, and Tj. After taking the logarithmic mapping on SE(3), the product is then mapped from the Lie algebra to the six-dimensional vector space (i.e., V mapping). Here, Ti is the UAV pose corresponding to the keyframe before the switch, Tj is the UAV pose corresponding to the current frame after the switch, and Tobs is the relative pose transformation given by the cross-view observation constraint.

[0063] In terms of joint state estimation and result output, the IMU pre-integration factor, visual reprojection factor, and cross-view constraint factor are added to the factor graph optimizer, and the weighted sum of squares of the residuals of each factor is used as the objective function, as shown in the following equation:

[0064] In the formula, The state vector to be estimated includes the position, velocity, attitude and inertial measurement unit zero bias of the UAV. The position component is in meters, the velocity component is in meters per second, the attitude component is in radians, and the zero bias component is in units according to the corresponding physical quantities. The value range is the real number of the corresponding physical quantity. In this embodiment, the estimated position components of a certain key frame are 1507 meters, 2003 meters and 498 meters. To estimate the optimal state vector that minimizes the objective function, the units and range of values ​​are the same. ; superscript Indicates vector transpose; , , These are the residual vectors for the IMU pre-integration factor, visual reprojection factor, and cross-view constraint factor, respectively. Their components are in units of the physical quantities of inertial pre-integration, pixel, and pose, respectively. The value range is the real number of the corresponding physical quantity and tends to 0 as the iteration converges. In this embodiment, each residual component tends to 0 after convergence. , , These are the observation covariance matrices for the IMU pre-integration factor, the visual reprojection factor, and the cross-view constraint factor, respectively. , , The information matrix is ​​the inverse of the corresponding covariance matrix, and its value range is a symmetric positive definite matrix. It is obtained by calibrating the noise parameters of each sensor. In this embodiment, the standard deviation of visual observation is taken as 1 pixel, and it is constructed accordingly. ; This represents the summation of all observations of various factors within the sliding window.

[0065] The objective function described above is solved using Gauss-Newton iteration with nonlinear least squares. The iteration terminates when the decrease in the objective function between two consecutive iterations is less than a preset convergence threshold or when a preset maximum number of iterations is reached. In this embodiment, the convergence threshold is set to... The maximum number of iterations is set to 20. The convergence threshold and the maximum number of iterations are determined based on the convergence curve of the objective function's descent with the number of iterations in the offline experiment. The descent that becomes gradual and the positioning accuracy no longer significantly improves with iterations is taken as the convergence threshold, and the corresponding number of iterations is taken as the maximum number of iterations, balancing positioning accuracy and real-time performance. The position, velocity, and attitude in the optimal state vector obtained by solving are used as the estimation results of the UAV at each key frame moment, and the fusion positioning result of the UAV is output accordingly.

[0066] Therefore, when feature degradation triggers channel switching, the factor graph simultaneously includes inertial constraints, visual constraints of the switched channel, and cross-view consistency constraints, so that the positioning results before and after the switch remain spatially continuous and consistent, achieving high-precision positioning of UAVs in feature degradation scenarios.

[0067] In summary, this invention discloses a UAV fusion localization method based on reverse perspective verification. It acquires multi-directional environmental image sequences synchronously collected by multi-directional visual sensors and motion state data synchronously collected by an inertial measurement unit. ORB feature extraction and tracking are performed on the multi-directional environmental image sequences, and degradation quality scores for each directional visual channel are calculated. Based on the degradation quality scores and degradation thresholds, the optimal backup visual channel is selected, and the primary localization channel is switched. During switching, cross-view feature matching is used to establish feature associations between the keyframe before and after the switch and the current frame after the switch for overlapping observation areas before and after the switch. Cross-view observation constraints are obtained through geometric constraint verification. Motion state data, the visual reprojection error of the currently active visual channel, and the cross-view observation constraints are uniformly input into a factor graph optimizer for joint state estimation, and the fusion localization result of the UAV is output. This invention improves the continuity of high-precision UAV localization in feature degradation scenarios by online evaluation and adaptive switching of the degradation state of multi-directional visual channels, and by tightly coupling and fusing the cross-view consistency constraints formed by reverse perspective verification of forward observations from the backward perspective at the switching moment with the factor graph.

[0068] The second embodiment of the present invention provides a UAV fusion positioning system based on reverse perspective verification, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the UAV fusion positioning system method based on reverse perspective verification as described above.

[0069] It should be noted that the UAV fusion positioning system based on reverse perspective verification provided in this embodiment of the invention is used to execute all the process steps of the UAV fusion positioning method based on reverse perspective verification in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0070] It should be noted that embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described above. The computer-readable storage medium is a read-only memory, a random access memory, a magnetic disk, an optical disk, or a USB flash drive.

[0071] It should be noted that the above system embodiments are merely illustrative. In physical implementation, the memory and processor are integrated into the same device or placed in two or more devices that work together through communication connections. Those skilled in the art can understand and implement these embodiments without any creative effort.

[0072] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A UAV fusion localization method based on reverse perspective verification, characterized in that, include: Acquire multi-directional environmental image sequences synchronously collected by the multi-directional vision sensors mounted on the UAV, as well as motion state data synchronously collected by the inertial measurement unit; ORB feature extraction and tracking are performed on the multi-directional environmental image sequence, and degradation quality scores for each directional visual channel are calculated. Based on the degradation quality score and the preset degradation threshold, the optimal backup visual channel is selected, and the positioning main channel is switched to the optimal backup visual channel. When switching visual channels, feature associations between keyframes before and after switching and the current frame after switching are established through cross-view feature matching for overlapping observation areas before and after switching, and geometric constraint verification is performed on the matching results to obtain cross-view observation constraints. The motion state data, the visual reprojection error of the currently active visual channel, and the cross-view observation constraints are uniformly input into the factor graph optimizer for joint state estimation, and the fused localization result of the UAV is output.

2. The UAV fusion positioning method based on reverse perspective verification according to claim 1, characterized in that, The acquisition of multi-directional environmental image sequences synchronously collected by the multi-directional vision sensor mounted on the UAV, and motion state data synchronously collected by the inertial measurement unit, includes: The multi-directional vision sensor includes a forward vision sensor facing the nose of the drone and a rearward vision sensor facing the tail. They simultaneously acquire environmental images of their respective directions and arrange the environmental images according to the acquisition time to obtain a multi-directional environmental image sequence. Based on the exposure time of each image frame in the multi-directional environmental image sequence, angular velocity and linear acceleration are collected by an inertial measurement unit and filtered, and motion state data are composed in timestamp order.

3. The UAV fusion positioning method based on reverse perspective verification according to claim 1, characterized in that, The step of performing ORB feature extraction and tracking on the multi-directional environmental image sequence and calculating the degradation quality score for each directional visual channel includes: ORB features are extracted from the current image frame of each visual channel, and the extracted ORB feature points are tracked with the previous image frame of the same visual channel to obtain the effective tracking feature points of each visual channel. The ratio of the number of effective tracked feature points to the preset number of reference features is used to obtain a tracking number score. Calculate the average disparity and disparity dispersion of the effective tracked feature points between the current image frame and the previous image frame, and obtain the disparity distribution score based on the average disparity and the disparity dispersion. Geometric consistency verification is performed on the effective tracking feature points, and a matching quality score is obtained based on the proportion of the number of verified interior points to the total number of effective tracking feature points. The degradation quality score for each visual channel is obtained by weighted summation of the tracking quantity score, the disparity distribution score, and the matching quality score.

4. The UAV fusion positioning method based on reverse perspective verification according to claim 3, characterized in that, The step of selecting the optimal backup visual channel based on the degradation quality score and a preset degradation threshold, and switching the primary positioning channel to the optimal backup visual channel, includes: When the degradation quality score of the currently active visual channel is lower than the degradation threshold, candidate channels with degradation quality scores higher than the degradation threshold are selected from visual channels in all directions other than the currently active visual channel. When the candidate channel exists, the candidate channel with the highest degradation quality score is determined as the optimal backup visual channel, and the positioning main channel is switched to the optimal backup visual channel. When no candidate channel exists, the currently active visual channel is maintained as the primary localization channel, and the confidence level of the fused localization result is marked as a downgraded state.

5. The UAV fusion positioning method based on reverse perspective verification according to claim 2, characterized in that, During visual channel switching, for the overlapping observation areas before and after the switching, feature associations are established between the keyframe before the switching and the current frame after the switching through cross-view feature matching, including: Based on the installation pose and field of view of the visual channel before and after switching on the UAV body, the overlap range of the field of view of the two visual channels is determined. When the keyframe before the switch and the current frame after the switch have the same view of the same spatial region, and the shared view region is located within the field of view overlap range, it is determined that there is an overlapping observation region before and after the switch. When the pre-switching visual channel is the visual channel corresponding to the forward visual sensor and the post-switching visual channel is the visual channel corresponding to the backward visual sensor, cross-view feature matching is performed on the overlapping observation area between the pre-switching keyframe acquired by the forward visual sensor and the post-switching current frame acquired by the backward visual sensor.

6. The UAV fusion positioning method based on reverse perspective verification according to claim 5, characterized in that, The geometric constraint verification of the matching results to obtain cross-view observation constraints includes: ORB feature descriptors are extracted from the keyframe before the switch and the current frame after the switch within the overlapping observation area; Calculate the matching distance between two sets of ORB feature descriptors, and retain candidate matching pairs whose matching distance is less than a preset descriptor distance threshold; The candidate matching pairs are verified by epipolar geometry constraints, and mismatched pairs that do not meet the epipolar geometry constraints are eliminated to obtain the verified cross-view matching pairs. Based on the verified cross-view matching pair, the relative pose transformation from the keyframe coordinate system before the switch to the current frame coordinate system after the switch is calculated, and the relative pose transformation is used as the cross-view observation constraint.

7. The UAV fusion positioning method based on reverse perspective verification according to claim 1, characterized in that, The process of inputting the motion state data, the visual reprojection error of the currently active visual channel, and the cross-view observation constraints into a unified input factor graph optimizer for joint state estimation, and outputting the fused localization result of the UAV, includes: Between adjacent keyframes of the currently active visual channel, the angular velocity and linear acceleration in the motion state data are pre-integrated to obtain the relative rotation increment, relative translation increment, and relative velocity increment, and the IMU pre-integration factor is constructed based on the relative rotation increment, the relative translation increment, and the relative velocity increment. Based on the observed pixel coordinates of each feature point in the currently active visual channel and the projected pixel coordinates of each feature point predicted based on the state vector to be estimated, the visual reprojection error of the currently active visual channel is calculated, and a visual reprojection factor is constructed based on the visual reprojection error. When switching visual channels, a cross-view constraint factor is constructed based on the cross-view observation constraint; the cross-view constraint factor is used to constrain the relative pose consistency between the keyframe before the switch and the current frame after the switch. The IMU pre-integration factor, the visual reprojection factor, and the cross-view constraint factor are added to the factor graph optimizer for joint state estimation, and the fused localization result is output.

8. The method according to claim 4, characterized in that, Also includes: After the positioning main channel is switched to the optimal backup visual channel, the degradation quality score of the original active visual channel before the switch is continuously monitored. When the degradation quality score of the original active visual channel rises above the degradation threshold and the difference between it and the degradation quality score of the optimal backup visual channel is greater than the preset recovery hysteresis threshold, the positioning main channel is switched back to the original active visual channel. The cross-view feature matching is re-executed on the overlapping observation regions before and after the cutback to obtain the cross-view observation constraints after the cutback, and the cross-view observation constraints after the cutback are input into the factor graph optimizer to participate in joint state estimation.

9. A UAV fusion positioning system based on reverse perspective verification, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the UAV fusion positioning method based on reverse perspective verification as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the UAV fusion positioning method based on reverse perspective verification as described in any one of claims 1 to 8.

Images