Visual navigation and positioning methods and systems for low-speed unmanned aerial vehicles

Abstract

This invention provides a visual navigation and positioning method and system for low-speed unmanned aerial vehicles (UAVs), comprising: Step S1: acquiring a current image through a visual sensor, preprocessing the acquired current image, and extracting feature points based on the preprocessed current image to obtain the image coordinate values ​​of the feature points; Step S2: obtaining attitude transformation matrix information in the inertial frame based on the position, attitude, and altitude information of the UAV measured by the inertial measurement unit and the altimeter, and subscribing to the attitude and altitude information output by the flight control system in the manner of attitude transformation matrix; Step S3: calculating the spatial position of the corresponding pixel in the world three-dimensional coordinate system based on the obtained image coordinate values ​​of the feature points and the obtained attitude and altitude information through a visual sensor model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a visual navigation and positioning method and system for low-speed unmanned aerial vehicles (UAVs). Primarily, it proposes an integrated hardware and software navigation and positioning solution based on a monocular vision sensor and a multi-mode positioning algorithm, specifically for small UAVs operating under global satellite navigation denial conditions. Background Technology

[0002] Small unmanned autonomous aerial vehicles (UAVs) are aircraft that fly under propulsion conditions, can perform multi-directional maneuvers, and can implement autonomous control strategies. Compared with large and medium-sized UAVs, they have lower speeds, making them better suited for demanding tasks such as search and surveillance, and observation of specific areas. However, in indoor environments or other conditions where global satellite navigation is denied, the aircraft cannot effectively obtain its own position information in three-dimensional space. Furthermore, if an inertial measurement unit (IMU) is used and its drift cannot be effectively suppressed, mission execution will be hindered.

[0003] Patent document CN113917939B (application number: 202111174390.0) discloses a positioning and navigation method, system, and computing device for an aircraft. The positioning and navigation method includes: obtaining real-time flight map data generated during the aircraft's current flight, the real-time flight map data including terrain data; determining whether the current flight area is a known area based on the terrain data in the real-time flight map data and pre-stored historical flight map data; if the current flight area is determined to be a known area, obtaining the aircraft's positioning and navigation data based on the pre-stored historical flight map data, and performing flight control based on the positioning and navigation data.

[0004] The visual navigation and positioning method in this invention has advantages such as strong autonomy and light load. It can serve as a relatively accurate feedback information for the control system. While correcting the measurement errors caused by inertial components, it can also make the autonomous flight control of the aircraft more accurate and stable. In addition, machine vision has sufficient technical support for feature and target recognition, which makes the solution more mature. The application of vision in aircraft control systems and related tasks has broad prospects and important research.

[0005] Therefore, this invention applies the homography relationship of visual imaging and combines it with the attitude information of the inertial measurement unit to provide an effective and feasible method for the autonomous positioning problem of small unmanned aerial vehicles under satellite navigation denied environment conditions. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a visual navigation and positioning method and system for low-speed unmanned aerial vehicles.

[0007] A visual navigation and positioning method for low-speed unmanned aerial vehicles provided by the present invention includes:

[0008] Step S1: Acquire the current image through the visual sensor, preprocess the acquired current image, and extract feature points based on the preprocessed current image to obtain the image coordinate values ​​of the feature points;

[0009] Step S2: Based on the aircraft's position, attitude, and altitude information measured by the aircraft's inertial measurement unit and altimeter, obtain the attitude transformation matrix information in the inertial frame, and subscribe to the attitude and altitude information output by the flight control in the form of attitude transformation matrix;

[0010] Step S3: Based on the obtained image coordinates of the feature points and the obtained pose and height information, the spatial position of the corresponding pixel in the world three-dimensional coordinate system is calculated through the visual sensor model.

[0011] Preferably, a monocular camera is strapped onto the aircraft fuselage at the required downward viewing angle, with a fixed mounting orientation of M. fix By combining the inertial measurement unit and altimeter inherent in the aircraft control system, an overall configuration scheme for the aircraft sensors is formed; the Zhang Zhengyou calibration method is used to perform strapdown installation calibration of the visual sensor to obtain the installation transformation matrix, intrinsic parameter matrix, and optical distortion matrix of the monocular camera relative to the aircraft body.

[0012] Preferably, the preprocessing of the acquired current image involves:

[0013] Step S1.1: Based on the optical distortion parameter matrix, the distortion of the current image is corrected by algorithm. The corrected image is separated in the HSV channel. Based on the characteristic that the brightness of the highlight area is different in different channels, the highlight area is segmented and the residual interference factors are eliminated through thresholding and erosion, thereby suppressing the highlight interference in the feature area.

[0014] Step S1.2: For regions of no interest or interference that appear during image processing, morphological dilation and erosion operations are used for filtering and elimination.

[0015] Preferably, step S1 employs:

[0016] Step S1.3: The image is processed using the Canny edge double threshold detection method. The high and low threshold ratio that meets the preset conditions is used. The gradient change of the image in two directions is calculated by the approximate limit difference of the first-order partial derivative of the Canny gradient operator. The maximum value of the gradient is analyzed by combining the direction of the gradient to determine the image edge.

[0017] Step S1.4: Perform Hough transform on the binary image with only edge information, store the information in the parameter space obtained by the Hough transform, and extract the required line slope and distance from the origin from the information in the parameter space; sort the line slope and distance from the origin with two parameters to separate line clusters, filter and extract the intersection points and feature lines in the environment, and then obtain the image coordinates of the feature points.

[0018] Preferably, step S2 involves: reading the aircraft position, attitude, and altitude information measured by the aircraft inertial measurement unit and the altimeter via serial communication, filtering the acquired aircraft position, attitude, and altitude information, and outputting the attitude transformation matrix information in the inertial frame; and subscribing to the attitude and altitude information output by the flight control system in the manner of the attitude transformation matrix.

[0019] Preferably, step S3 involves: for structured environments with regular straight line features and unstructured environments with irregular field of view features in the downward field of view, based on the obtained image coordinate values ​​of feature points and the obtained pose and height information, using different positioning algorithms to calculate the spatial position of the corresponding pixel points in the world three-dimensional coordinate system;

[0020] The structured environment of the regularized straight line features is based on the image coordinates X of the feature points. uv and the world coordinates X of feature points calibrated in a structured environment W Based on the measured world coordinates of the extracted feature points Deviation ΔX relative to the actual value W Regarding the position X of the aircraft C The data is then corrected, and the corrected data participates in the iterative calculation of the next position calculation algorithm. The main model of the iterative algorithm is as follows:

[0021]

[0022]

[0023]

[0024] Where s represents the scale factor, which is the ratio between the camera's logical resolution and its physical resolution; This represents the transformation matrix from the camera coordinate system to the world coordinate system for the k-th frame image, obtained based on pose and height information. The inverse matrix representing the intrinsic parameter matrix of the camera;

[0025] The unstructured environment with irregular field of view features, based on the feature point image coordinate pairs p1 and p2 matched between two adjacent frames, obtains the rotation matrix R and translation vector T for inter-frame transformation by solving the fundamental matrix F and essential matrix E in epipolar geometry. Then, pose and height information are used to verify and compensate the rotation matrix R and translation vector T, restoring the camera's motion state in three-dimensional space. Finally, coordinate transformation M with fixed pose relationships is performed. fix Obtain the navigation and positioning information of the aircraft.

[0026] A visual navigation and positioning system for a low-speed unmanned aerial vehicle provided by the present invention includes:

[0027] Module M1: Acquires the current image through a vision sensor, preprocesses the acquired current image, and extracts feature points based on the preprocessed current image to obtain the image coordinate values ​​of the feature points;

[0028] Module M2: Based on the aircraft's position, attitude, and altitude information measured by the aircraft's inertial measurement unit and altimeter, it obtains the attitude transformation matrix information in the inertial frame and subscribes to the attitude and altitude information output by the flight control in the form of attitude transformation matrix.

[0029] Module M3: Based on the obtained image coordinates of feature points and the obtained pose and height information, the spatial position of the corresponding pixel in the world three-dimensional coordinate system is calculated through the visual sensor model.

[0030] Preferably, a monocular camera is strapped onto the aircraft fuselage at the required downward viewing angle, with a fixed mounting orientation of M. fix By combining the inertial measurement unit and altimeter inherent in the aircraft control system, an overall configuration scheme for the aircraft sensors is formed; the Zhang Zhengyou calibration method is used to perform strapdown installation calibration of the visual sensor to obtain the installation transformation matrix, intrinsic parameter matrix, and optical distortion matrix of the monocular camera relative to the aircraft body.

[0031] Preferably, the preprocessing of the acquired current image involves:

[0032] Module M1.1: Based on the optical distortion parameter matrix, the distortion of the current image is corrected by algorithm. The corrected image is separated in the HSV channel. Based on the characteristic that the brightness of the highlight area is different in different channels, the highlight area is segmented and the residual interference factors are eliminated through thresholding and erosion, thereby suppressing the highlight interference in the feature area.

[0033] Module M1.2: For regions of no interest or interference that appear during image processing, morphological dilation and erosion operations are used to filter and eliminate them;

[0034] Module M1.3: The image is processed using the Canny edge double threshold detection method. The high and low threshold ratio that meets the preset conditions is used. The gradient change of the image in two directions is calculated by the approximate limit difference of the first-order partial derivative of the Canny gradient operator. The maximum value of the gradient is analyzed by combining the gradient direction to determine the image edge.

[0035] Module M1.4: Performs Hough transform on a binary image containing only edge information, stores the information in the parameter space obtained by the Hough transform, and extracts the required line slope and distance from the origin from the information in the parameter space; performs two-parameter sorting on the line slope and distance from the origin to separate line clusters, filters and extracts intersections and feature lines in the environment, and then obtains the image coordinates of feature points.

[0036] Preferably, module M3 employs the following approach: for structured environments with regular straight-line features and unstructured environments with irregular field-of-view features in the downward field of view, based on the obtained image coordinate values ​​of feature points and the obtained pose and height information, different positioning algorithms are used to calculate the spatial position of the corresponding pixel in the world three-dimensional coordinate system.

[0037] The structured environment of the regularized straight line features is based on the image coordinates X of the feature points. uv and the world coordinates X of feature points calibrated in a structured environment W Based on the measured world coordinates of the extracted feature points Deviation ΔX relative to the actual value W Regarding the position X of the aircraft C The data is then corrected, and the corrected data participates in the iterative calculation of the next position calculation algorithm. The main model of the iterative algorithm is as follows:

[0038]

[0039]

[0040]

[0041] Where s represents the scale factor, which is the ratio between the camera's logical resolution and its physical resolution; This represents the transformation matrix from the camera coordinate system to the world coordinate system for the k-th frame image, obtained based on pose and height information. The inverse matrix representing the intrinsic parameter matrix of the camera;

[0042] The unstructured environment with irregular field of view features, based on the feature point image coordinate pairs p1 and p2 matched between two adjacent frames, obtains the rotation matrix R and translation vector T for inter-frame transformation by solving the fundamental matrix F and essential matrix E in epipolar geometry. Then, pose and height information are used to verify and compensate the rotation matrix R and translation vector T, restoring the camera's motion state in three-dimensional space. Finally, coordinate transformation M with fixed pose relationships is performed. fix Obtain the navigation and positioning information of the aircraft.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. This invention provides an autonomous navigation and positioning method based on the aircraft's own inertial measurement unit and monocular camera. Compared with the traditional navigation method that relies solely on satellite navigation system or inertial measurement unit, this method does not depend on external sensor information of the aircraft, thus ensuring the autonomy of the aircraft.

[0045] 2. The software algorithm of this invention is easy to implement and can guarantee real-time performance on embedded platforms. It is suitable for rapidly changing mission environments of aircraft. At the same time, the algorithm is easy to embed into flight control programs and can support flight control in suppressing the cumulative error of the inertial measurement unit, so that the navigation and positioning information error does not gradually diverge over time.

[0046] 3. The hardware configuration of this invention is simple. According to the actual mission requirements of the aircraft, a monocular camera can be configured and strapped to the aircraft.

[0047] 4. The overall navigation scheme of the present invention has strong hardware and software scalability. In addition to being applicable to low-speed aircraft, it can also be extended to other intelligent agents or robots with autonomous movement.

[0048] 5. The overall solution of the present invention has a low cost, does not have high requirements for the performance of the imaging sensor, and the communication interface can adopt a common standard protocol, making it suitable for aircraft designs with limited costs. Attached Figure Description

[0049] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0050] Figure 1 This is a flowchart of the visual navigation algorithm for unmanned aerial vehicles. Detailed Implementation

[0051] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0052] The problem to be solved by this invention is to realize and improve the stability and robustness of the navigation and positioning algorithm for small aircraft based on monocular vision measurement and aircraft attitude calculation model, combined with optimized processing of aircraft images.

[0053] To solve the technical problem, the technical solution adopted by this invention is: a hardware and software integrated navigation and positioning solution based on a monocular vision sensor and a multi-mode positioning algorithm.

[0054] Example 1

[0055] According to the present invention, a visual navigation and positioning method for low-speed unmanned aerial vehicles is provided, such as... Figure 1 As shown, it includes:

[0056] Step S1: Acquire the current image through the visual sensor, preprocess the acquired current image, and extract feature points based on the preprocessed current image to obtain the image coordinate values ​​of the feature points;

[0057] Step S2: Based on the aircraft's position, attitude, and altitude information measured by the aircraft's inertial measurement unit and altimeter, obtain the attitude transformation matrix information in the inertial frame, and subscribe to the attitude and altitude information output by the flight control in the form of attitude transformation matrix;

[0058] Step S3: Based on the obtained image coordinates of the feature points and the obtained pose and height information, calculate the spatial position of the corresponding pixel in the world three-dimensional coordinate system using the visual sensor model;

[0059] The vision sensor model

[0060] Specifically, a monocular camera is strapped onto the aircraft fuselage at the required downward viewing angle, with a fixed installation orientation of M. fix By combining the inertial measurement unit and altimeter inherent in the aircraft control system, an overall sensor configuration scheme for the aircraft is formed; the Zhang Zhengyou calibration method is used to perform strapdown installation calibration of the vision sensor, obtaining the installation transformation matrix and intrinsic parameter matrix M of the monocular camera relative to the aircraft body. in In addition to basic parameters such as the optical distortion matrix, the video stream information captured by the camera and the measurement sensor information of the flight control system are fused together in the flight control computer, and the navigation information is output through algorithm calculation.

[0061] Specifically, the preprocessing of the acquired current image involves:

[0062] Step S1.1: Based on the optical distortion parameter matrix, the distortion of the current image is corrected by algorithm. The corrected image is separated in the HSV channel. Based on the characteristic that the brightness of the highlight area is different in different channels, the highlight area is segmented and the residual interference factors are eliminated through thresholding and erosion, thereby suppressing the highlight interference in the feature area.

[0063] Step S1.2: For regions of no interest or interference that appear during image processing, morphological dilation and erosion operations are used for filtering and elimination.

[0064] Specifically, step S1 employs the following:

[0065] Step S1.3: The image is processed using the Canny edge double threshold detection method. The high and low threshold ratios are between 2:1 and 3:1. The gradient changes of the image in two directions are calculated by approximate limiting difference of the first-order partial derivative of the Canny gradient operator. The maximum value of the gradient is analyzed by combining the gradient direction to determine the image edge. The accurate extraction of edge information simplifies the design of the subsequent line fitting algorithm and reduces the error of feature point detection in the image.

[0066] Step S1.4: Perform a Hough transform on the binary image containing only edge information, store the parameter space information obtained from the Hough transform, and extract the required line slope and distance from the origin from the parameter space information; perform dual-parameter sorting on the line slope and distance from the origin to separate line clusters, filter and extract intersections and feature lines in the environment, and then obtain the image coordinates of feature points. For unstructured environments, extract SIFT features or ORB features between adjacent frames, and complete matching and mismatch processing.

[0067] For image sequences acquired by vision sensors, a combination of preprocessing methods such as optical distortion correction, specular suppression, and morphological filtering, as well as feature extraction algorithms such as improved Hough transform, edge detection, and fast corner feature extraction, are used to ensure real-time image processing and robust, high-precision information output.

[0068] Specifically, step S2 involves: reading the aircraft position, attitude, and altitude information measured by the aircraft inertial measurement unit and the altimeter via serial communication, filtering the acquired aircraft position, attitude, and altitude information, and outputting the attitude transformation matrix information in the inertial frame; and subscribing to the attitude and altitude information output by the flight control system in the manner of the attitude transformation matrix.

[0069] Specifically, step S3 employs the following approach: for structured environments with regular straight line features and unstructured environments with irregular field of view features in the downward field of view, based on the obtained image coordinate values ​​of feature points and the obtained pose and height information, different positioning algorithms are used to calculate the spatial position of the corresponding pixel points in the world three-dimensional coordinate system.

[0070] The structured environment of the regularized straight line features is based on the image coordinates X of the feature points. uv and the world coordinates X of feature points calibrated in a structured environment W Based on the measured world coordinates of the extracted feature points Deviation ΔX relative to the actual value W Regarding the position X of the aircraft C The data is then corrected, and the corrected data participates in the iterative calculation of the next position calculation algorithm to ensure that the navigation and positioning information does not accumulate and drift with increasing flight time; the main model of the iterative algorithm is as follows:

[0071]

[0072]

[0073]

[0074]

[0075] Where s represents the scale factor, which is the ratio between the camera's logical resolution and its physical resolution; This represents the transformation matrix from the camera coordinate system to the world coordinate system for the k-th frame image; The inverse matrix representing the intrinsic parameter matrix of the camera;

[0076] The unstructured environment with irregular field of view features, based on the feature point image coordinate pairs p1 and p2 matched between two adjacent frames, obtains the rotation matrix R and translation vector T for inter-frame transformation by solving the fundamental matrix F and essential matrix E in epipolar geometry. Then, pose and height information are used to verify and compensate the rotation matrix R and translation vector T, restoring the camera's motion state in three-dimensional space. Finally, coordinate transformation M with fixed pose relationships is performed. fix Obtain the navigation and positioning information of the aircraft.

[0077] A visual navigation and positioning system for a low-speed unmanned aerial vehicle provided by the present invention includes:

[0078] Module M1: Acquires the current image through a vision sensor, preprocesses the acquired current image, and extracts feature points based on the preprocessed current image to obtain the image coordinate values ​​of the feature points;

[0079] Module M2: Based on the aircraft's position, attitude, and altitude information measured by the aircraft's inertial measurement unit and altimeter, it obtains the attitude transformation matrix information in the inertial frame and subscribes to the attitude and altitude information output by the flight control in the form of attitude transformation matrix.

[0080] Module M3: Based on the obtained image coordinates of feature points and the obtained pose and height information, the spatial position of the corresponding pixel in the world three-dimensional coordinate system is calculated through the visual sensor model;

[0081] The vision sensor model

[0082] Specifically, a monocular camera is strapped onto the aircraft fuselage at the required downward viewing angle, with a fixed installation orientation of M. fix By combining the inertial measurement unit and altimeter inherent in the aircraft control system, an overall sensor configuration scheme for the aircraft is formed; the Zhang Zhengyou calibration method is used to perform strapdown installation calibration of the vision sensor, obtaining the installation transformation matrix and intrinsic parameter matrix M of the monocular camera relative to the aircraft body. in In addition to basic parameters such as the optical distortion matrix, the video stream information captured by the camera and the measurement sensor information of the flight control system are fused together in the flight control computer, and the navigation information is output through algorithm calculation.

[0083] Specifically, the preprocessing of the acquired current image involves:

[0084] Module M1.1: Based on the optical distortion parameter matrix, the distortion of the current image is corrected by algorithm. The corrected image is separated in the HSV channel. Based on the characteristic that the brightness of the highlight area is different in different channels, the highlight area is segmented and the residual interference factors are eliminated through thresholding and erosion, thereby suppressing the highlight interference in the feature area.

[0085] Module M1.2: For regions of no interest or interference that appear during image processing, morphological dilation and erosion operations are used to filter and eliminate them.

[0086] Specifically, module M1 adopts:

[0087] Module M1.3: The Canny edge detection method with dual thresholds is used to process the image. The high and low threshold ratios are between 2:1 and 3:1. The gradient changes of the image in two directions are calculated by approximate limiting difference of the first-order partial derivative of the Canny gradient operator. The gradient maxima are analyzed by combining the gradient direction to determine the image edges. The accurate extraction of edge information simplifies the design of the subsequent line fitting algorithm and reduces the error of feature point detection in the image.

[0088] Module M1.4: Performs Hough Transform on binary images containing only edge information, stores the parameter space information obtained from the Hough Transform, and extracts the required line slopes and distances from the origin from the parameter space information. It then performs two-parameter sorting on the line slopes and distances from the origin to separate line clusters, filters and extracts intersections and feature lines in the environment, and obtains the image coordinates of feature points. For unstructured environments, it extracts SIFT or ORB features between adjacent frames and performs matching and mismatch processing.

[0089] For image sequences acquired by vision sensors, a combination of preprocessing methods such as optical distortion correction, specular suppression, and morphological filtering, as well as feature extraction algorithms such as improved Hough transform, edge detection, and fast corner feature extraction, are used to ensure real-time image processing and robust, high-precision information output.

[0090] Specifically, module M2 adopts the following approach: reading the aircraft position, attitude, and altitude information measured by the aircraft inertial measurement unit and altimeter via serial communication, filtering the acquired aircraft position, attitude, and altitude information, and outputting the attitude transformation matrix information in the inertial frame; and subscribing to the attitude and altitude information output by the flight control system in the form of the attitude transformation matrix.

[0091] Specifically, module M3 employs the following approach: for structured environments with regular straight-line features and unstructured environments with irregular field-of-view features in the downward field of view, based on the obtained image coordinate values ​​of feature points and the obtained pose and height information, different positioning algorithms are used to calculate the spatial position of the corresponding pixel points in the world three-dimensional coordinate system.

[0092] The structured environment of the regularized straight line features is based on the image coordinates X of the feature points. uv and the world coordinates X of feature points calibrated in a structured environment W Based on the measured world coordinates of the extracted feature points Deviation ΔX relative to the actual value W Regarding the position X of the aircraft C The data is then corrected, and the corrected data participates in the iterative calculation of the next position calculation algorithm to ensure that the navigation and positioning information does not accumulate and drift with increasing flight time; the main model of the iterative algorithm is as follows:

[0093]

[0094]

[0095]

[0096] Where s represents the scale factor, which is the ratio between the camera's logical resolution and its physical resolution; This represents the transformation matrix from the camera coordinate system to the world coordinate system for the k-th frame image; The inverse matrix representing the intrinsic parameter matrix of the camera;

[0097] The irregular, unstructured environment of the field of view features, based on the feature point image coordinate pairs p1 and p2 matched between two adjacent frames, obtains the rotation matrix R and translation vector T for inter-frame transformation by solving the fundamental matrix F and essential matrix E in epipolar geometry. Values ​​such as the attitude angle change rate, position change rate, and altitude change rate of the altimeter can be used. Based on the continuity of the aircraft's spatial motion, i.e., the change rate will not undergo abrupt changes exceeding a set threshold, the rotation matrix R and translation vector T are verified and compensated. Simultaneously, inaccurate data from a certain frame is filtered out, thereby better restoring the camera's motion state in three-dimensional space. Furthermore, a coordinate transformation M based on a fixed pose relationship is then performed. fix Obtain the navigation and positioning information of the aircraft.

[0098] Example 2

[0099] Example 2 is a preferred example of Example 1.

[0100] A visual navigation and positioning method for low-speed unmanned aerial vehicles provided by the present invention includes:

[0101] Step one: Regarding hardware configuration, taking a conventional low-speed quadcopter drone as an example, based on the drone's mission requirements, select a camera that meets the performance requirements such as field of view, resolution, and frame rate. The camera, as a sensor, is strapped onto the drone's frame and installed at a certain downward angle according to the information to be collected (the ground). The fixed pose transformation relationship between the camera and the aircraft's coordinate system is measured and obtained as M. fix The camera can be connected to the aircraft's flight control computer via standard interfaces such as USB. Together with the aircraft's inherent inertial measurement unit and altimeter, they form the overall sensor configuration scheme.

[0102] Step two: To support subsequent algorithm calculations, the camera's intrinsic parameter matrix needs to be calibrated. The main method is the Zhang Zhengyou calibration method, which involves acquiring several checkerboard images of the camera for calibration. This yields information such as the camera's optical center shift, focal length, and distortion parameters, which in turn provides the camera's intrinsic parameter matrix M. in And optical distortion matrix, etc.

[0103] Step three involves fusing the video stream information captured by the camera and the measurement sensor information from the flight control system within the flight control computer. The algorithm then calculates and outputs the navigation information. Taking a typical quadcopter as an example, the measurement sensors in the flight control system include ultrasonic altimeters, inertial measurement units (IMUs), geomagnetic direction finders, and optical flow velocity sensors. The algorithm typically uses the measurement information from the altimeter and the IMU. The IMU is generally connected to the flight control computer via a UART serial port, where it filters and integrates the data within the flight control software before outputting the aircraft's attitude angles and position information.

[0104] Step four involves preprocessing the video sequence acquired by the sensor. The main purpose is to eliminate interference factors such as highlights and complex textures, and to focus on the region of interest (ROI) to improve the overall real-time processing. For highlight effects such as ground reflections, image separation in the HSV channels is typically performed. Based on the significant grayscale differences in highlight areas across different channels, these areas are segmented using grayscale histograms. Following channel separation, thresholding and morphological erosion are used for fine-tuning. Uninteresting or interfering regions encountered during image processing can be filtered out using morphological dilation and erosion operations. The specific filtering parameters can be determined through desktop simulation experiments.

[0105] Step 5 involves feature extraction from the image. Following the design principles of the multi-mode algorithm, for known environments such as indoors, feature points with significant gradient changes, large differences in grayscale values, and locations at line intersections can be extracted from the known field of view. To accurately extract features for subsequent calculations, the Canny edge detection method with dual thresholds is first used to process the image. Desktop simulation determines a high-low threshold ratio between 2:1 and 3:1. The gradient changes in two directions are calculated using the approximate difference of the first-order partial derivative of the Canny gradient operator. Then, the gradient maxima are analyzed based on the gradient direction to determine the image edges. Next, a Hough transform operation is performed on the binary image containing only edge information to extract line information. Generally, the Hough transform has a large error in fitting lines and exhibits the phenomenon of multiple line clusters overlapping, which is not conducive to accurately calculating intersection points. Therefore, the Hough transform is improved by storing the parameter space information obtained from the Hough transform and extracting the required line slope and distance from the origin from the parameter space. Two parameters are sorted using a two-parameter algorithm to separate line clusters, and then intersection points in the environment are filtered and extracted. The pixel value of the intersection point is the image coordinate of the feature point. For unknown unstructured environments, SIFT or ORB feature extraction algorithms are used to extract SIFT or ORB feature points between two adjacent frames and perform inter-frame matching. RANSAC algorithm is used to remove mismatched points, thus obtaining the pixel coordinates of the same feature point in two different frames, which can be used as input for subsequent algorithms.

[0106] Step six: Integrate information from all sensors to perform algorithm calculations. Under conditions where the environment is known or has distinct features (such as the intersection of straight lines on the ground), the algorithm uses the attitude transformation matrix to transform from the camera coordinate system to the world coordinate system. The method involves subscribing to data using altitude information Z from the aircraft's altimeter sensor, based on the aforementioned feature point pixel coordinates X. uv and the world coordinates X of feature points in a known environment W Based on the established homography relation formula, the camera's position X in the world coordinate system can be calculated. C Based on the measured world coordinates of the extracted feature points. Deviation ΔX relative to the actual value W Regarding the position X of the aircraft C The data is then corrected, and the corrected data participates in the iterative calculation of the next position calculation algorithm. The main model of the iterative algorithm is as follows, where the superscript k represents the image of the kth frame.

[0107]

[0108]

[0109]

[0110] Step 5: Under unknown environmental conditions, based on the feature point pairs p1 and p2 matched in two adjacent frames, the following relationship exists:

[0111] x2=Rx1+T

[0112] Where R and T are the rotation matrix and translation vector, respectively, for the transformation between two adjacent frames.

[0113]

[0114]

[0115] This represents the continuous movement of the camera, and consequently, the continuous movement of the aircraft. Based on the well-established epipolar constraint relationship in the field of computer vision, we have:

[0116]

[0117] By solving the fundamental matrix F and essential matrix E in epipolar geometry using a mature algorithm, the motion state of the camera in three-dimensional space is recovered. Then, a coordinate transformation M with a fixed pose relationship is performed. fix Obtain the navigation and positioning information of the aircraft.

[0118] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0119] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A visual navigation and positioning method for low-speed unmanned aerial vehicles, characterized in that, include: Step S1: Acquire the current image through the visual sensor, preprocess the acquired current image, and extract feature points based on the preprocessed current image to obtain the image coordinate values ​​of the feature points; Step S2: Based on the aircraft's position, attitude, and altitude information measured by the aircraft's inertial measurement unit and altimeter, obtain the attitude transformation matrix information in the inertial frame, and subscribe to the attitude and altitude information output by the flight control in the form of attitude transformation matrix; Step S3: Based on the obtained image coordinates of the feature points and the obtained pose and height information, calculate the spatial position of the corresponding pixel in the world three-dimensional coordinate system through the visual sensor model; The visual sensor model is a mathematical model for obtaining the spatial coordinates of a target point through a two-dimensional image. The mathematical model is a mathematical transformation relationship between the image pixel coordinates and the world coordinates of the target point, established under the consideration of lens distortion. A monocular camera is strapped on the aircraft fuselage at the required downward viewing angle, with a fixed mounting posture relationship as follows: By combining the inertial measurement unit and altimeter inherent in the aircraft control system, an overall configuration scheme for the aircraft sensors is formed; the Zhang Zhengyou calibration method is used to perform strapdown installation calibration of the visual sensor to obtain the installation transformation matrix, intrinsic parameter matrix, and optical distortion matrix of the monocular camera relative to the aircraft body; Step S3 employs the following approach: For structured environments with regular straight line features and unstructured environments with irregular field of view features in the downward field of view, different positioning algorithms are used to calculate the spatial position of the corresponding pixel in the world three-dimensional coordinate system based on the obtained image coordinate values ​​of the feature points and the obtained pose and height information. The structured environment of the regularized straight line features is based on the image coordinates of the feature points. and the world coordinates of feature points calibrated in a structured environment Based on the measured world coordinates of the extracted feature points Deviation from actual value Regarding the aircraft's position The data is then corrected, and the corrected data participates in the iterative calculation of the next position calculation algorithm. The main model of the iterative algorithm is as follows: (1) (2) Where s represents the scale factor, which is the ratio between the camera's logical resolution and its physical resolution; This represents the transformation matrix from the camera coordinate system to the world coordinate system for the k-th frame image, obtained based on pose and height information. The inverse matrix representing the camera's intrinsic parameter matrix; The unstructured environment with irregular field of view features is based on the feature point image coordinate pairs matched between two adjacent frames. and By solving the fundamental matrix in epipolar geometry With the essential matrix Obtain the rotation matrix of the inter-frame transformation With translation vector And use attitude and height information to adjust the rotation matrix. With translation vector Verification and compensation are performed to restore the camera's motion state in three-dimensional space, and then coordinate transformation is performed using a fixed pose relationship. Obtain the navigation and positioning information of the aircraft.

2. The visual navigation and positioning method for low-speed unmanned aerial vehicles according to claim 1, characterized in that, The preprocessing of the acquired current image involves: Step S1.1: Based on the optical distortion parameter matrix, the distortion of the current image is corrected by algorithm. The corrected image is separated in the HSV channel. Based on the characteristic that the brightness of the highlight area is different in different channels, the highlight area is segmented and the residual interference factors are eliminated through thresholding and erosion, thereby suppressing the highlight interference in the feature area. Step S1.2: For regions of no interest or interference that appear during image processing, morphological dilation and erosion operations are used for filtering and elimination.

3. The visual navigation and positioning method for low-speed unmanned aerial vehicles according to claim 1, characterized in that, Step S1 adopts the following: Step S1.3: The image is processed using the Canny edge double threshold detection method. The high and low threshold ratio that meets the preset conditions is used. The gradient change of the image in two directions is calculated by the approximate limit difference of the first-order partial derivative of the Canny gradient operator. The maximum value of the gradient is analyzed by combining the direction of the gradient to determine the image edge. Step S1.4: Perform Hough transform on the binary image with only edge information, store the information in the parameter space obtained by the Hough transform, and extract the required line slope and distance from the origin from the information in the parameter space; sort the line slope and distance from the origin with two parameters to separate line clusters, filter and extract the intersection points and feature lines in the environment, and then obtain the image coordinates of the feature points.

4. The visual navigation and positioning method for low-speed unmanned aerial vehicles according to claim 1, characterized in that, Step S2 involves: reading the aircraft's position, attitude, and altitude information measured by the aircraft's inertial measurement unit and altimeter via serial communication; filtering the acquired aircraft position, attitude, and altitude information; and outputting the attitude transformation matrix information in the inertial frame. The attitude and altitude information output by the flight control system are then subscribed to using the attitude transformation matrix.

5. A visual navigation and positioning system for a low-speed unmanned aerial vehicle, characterized in that, include: Module M1: Acquires the current image through a vision sensor, preprocesses the acquired current image, and extracts feature points based on the preprocessed current image to obtain the image coordinate values ​​of the feature points; Module M2: Based on the aircraft's position, attitude, and altitude information measured by the aircraft's inertial measurement unit and altimeter, it obtains the attitude transformation matrix information in the inertial frame and subscribes to the attitude and altitude information output by the flight control in the form of attitude transformation matrix. Module M3: Based on the obtained image coordinates of feature points and the obtained pose and height information, the spatial position of the corresponding pixel in the world three-dimensional coordinate system is calculated through the visual sensor model; The visual sensor model is a mathematical model for obtaining the spatial coordinates of a target point through a two-dimensional image. The mathematical model is a mathematical transformation relationship between the image pixel coordinates and the world coordinates of the target point, established under the consideration of lens distortion. A monocular camera is strapped on the aircraft fuselage at the required downward viewing angle, with a fixed mounting posture relationship as follows: By combining the inertial measurement unit and altimeter inherent in the aircraft control system, an overall configuration scheme for the aircraft sensors is formed; the Zhang Zhengyou calibration method is used to perform strapdown installation calibration of the visual sensor to obtain the installation transformation matrix, intrinsic parameter matrix, and optical distortion matrix of the monocular camera relative to the aircraft body; The module M3 adopts the following approach: for structured environments with regular straight line features and unstructured environments with irregular field of view features in the downward field of view, based on the obtained image coordinate values ​​of feature points and the obtained pose and height information, different positioning and calculation algorithms are used to calculate the spatial position of the corresponding pixel points in the world three-dimensional coordinate system. The structured environment of the regularized straight line features is based on the image coordinates of the feature points. and the world coordinates of feature points calibrated in a structured environment Based on the measured world coordinates of the extracted feature points Deviation from actual value Regarding the aircraft's position The data is then corrected, and the corrected data participates in the iterative calculation of the next position calculation algorithm. The main model of the iterative algorithm is as follows: (1) (2) Where s represents the scale factor, which is the ratio between the camera's logical resolution and its physical resolution; This represents the transformation matrix from the camera coordinate system to the world coordinate system for the k-th frame image, obtained based on pose and height information. The inverse matrix representing the camera's intrinsic parameter matrix; The unstructured environment with irregular field of view features is based on the feature point image coordinate pairs matched between two adjacent frames. and By solving the fundamental matrix in epipolar geometry With the essential matrix Obtain the rotation matrix of the inter-frame transformation With translation vector And use attitude and height information to adjust the rotation matrix. With translation vector Verification and compensation are performed to restore the camera's motion state in three-dimensional space, and then coordinate transformation is performed using a fixed pose relationship. Obtain the navigation and positioning information of the aircraft.

6. The low-speed unmanned aerial vehicle visual navigation and positioning system according to claim 5, characterized in that, The preprocessing of the acquired current image involves: Module M1.1: Based on the optical distortion parameter matrix, the distortion of the current image is corrected by algorithm. The corrected image is separated in the HSV channel. Based on the characteristic that the brightness of the highlight area is different in different channels, the highlight area is segmented and the residual interference factors are eliminated through thresholding and erosion, thereby suppressing the highlight interference in the feature area. Module M1.2: For regions of no interest or interference that appear during image processing, morphological dilation and erosion operations are used to filter and eliminate them; Module M1.3: The image is processed using the Canny edge double threshold detection method. The high and low threshold ratio that meets the preset conditions is used. The gradient change of the image in two directions is calculated by the approximate limit difference of the first-order partial derivative of the Canny gradient operator. The maximum value of the gradient is analyzed by combining the gradient direction to determine the image edge. Module M1.4: Performs Hough transform on a binary image containing only edge information, stores the information in the parameter space obtained by the Hough transform, and extracts the required line slope and distance from the origin from the information in the parameter space; performs two-parameter sorting on the line slope and distance from the origin to separate line clusters, filters and extracts intersections and feature lines in the environment, and then obtains the image coordinates of feature points.

Images