Dynamic feature point elimination method based on normal distribution

Abstract

The invention discloses a dynamic feature point elimination method based on normal distribution. The method comprises the following steps: constructing an image feature point set; acquiring a relative rotation matrix between two adjacent frames of images; acquiring image feature point information of each feature point of the first frame of image in the continuous n frames of images; if the unmanned equipment is in a posture-invariant state and the current feature point of the first frame of image is in the image feature point set, obtaining correction pixel coordinates of the current feature point in each frame of image except the first frame of image based on the relative rotation matrix; and acquiring a pixel coordinate difference value of each feature point in the first frame of image and the Mth frame of image, and removing the feature points of which the pixel coordinate difference values are greater than three times of the standard deviation. According to the method, the posture of the feature point is continuously corrected based on the relative rotation matrix so as to eliminate the influence of the posture change of the unmanned equipment on the posture of the feature point, and finally, the dynamic point is removed based on the pixel coordinate difference value, so that the removal of the dynamic feature point can be quickly completed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for eliminating dynamic feature points based on normal distribution. Background Art

[0002] In the absence of a GPS signal or when the GPS signal is interfered with, unmanned devices (such as drones) can use a VIO (Visual Inertial Odometry) program to complete positioning and direction estimation by fusing a visual camera with an inertial sensor (IMU). This process requires solving the unmanned device's posture based on visual feature point matching and global optimization. It mainly includes the following steps: first, feature points are extracted from the image. The matching feature points between the two frames are used to calculate the posture change of the shooting device (the camera on the unmanned device) when the two frames were captured. This posture change can include information such as posture change and angle change, and the unmanned device is located based on this posture change.

[0003] However, in some scenes there may be dynamically moving objects, and the extracted feature points may be on the dynamic objects. In this case, visual constraints cannot provide reliable observation values, resulting in large errors, which affects the calculation accuracy of the pose. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a dynamic feature point elimination method based on normal distribution, which continuously corrects the feature point posture based on the relative rotation matrix to eliminate the influence of the unmanned equipment's own posture changes on the feature point posture, and finally eliminates dynamic points based on the normal distribution principle and pixel coordinate difference, thereby quickly completing the removal of dynamic feature points to ensure the accuracy of subsequent posture calculation results.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for eliminating dynamic feature points based on normal distribution is provided, which includes the following steps:

[0007] Constructing an image feature point set, wherein the set includes a plurality of image feature point information, and each image feature point information includes an image feature point index ID and the pixel coordinates of the image feature point in each frame of consecutive i frames;

[0008] Get the relative rotation matrix R_P_C between two adjacent frames of image;

[0009] Obtain image feature point information for each feature point in the first frame of n consecutive image frames, and determine whether the unmanned device is in a state of unchanged posture when acquiring n frames of image frames; if it is in a state of unchanged posture, and the current feature point of the first frame of image is in the image feature point set, obtain the corrected pixel coordinates of the current feature point in each frame of image except the first frame of image based on the relative rotation matrix R_P_C, and replace the pixel coordinates of the current feature point in each frame of image except the first frame of image in the image feature point set with the corresponding corrected pixel coordinates;

[0010] Also, when the unmanned equipment obtains M consecutive frames of images without any posture change, the pixel coordinate difference of each feature point in the 1st frame image and the Mth frame image is obtained according to the pixel coordinates of the feature points stored in the current image feature point set, and the mean and standard deviation of all pixel coordinate differences are calculated, and feature points with pixel coordinate differences greater than three times the standard deviation are eliminated.

[0011] Preferably, the image feature point set is of Map data type, which stores index ID and pixel coordinates as keys and values respectively.

[0012] Preferably, obtaining the relative rotation matrix R_P_C between two adjacent frames of image includes the following steps:

[0013] Receive m consecutive frames of images and IMU data through the VIO system, and obtain the average angular velocity based on the IMU data received by the VIO system during the time period between two adjacent frames of images arriving at the VIO system;

[0014] The average angular velocity is converted from the IMU coordinate system to the camera coordinate system, and the relative rotation matrix R_P_C is calculated.

[0015] Preferably, the average angular velocity is obtained according to formula (1):

[0016]

[0017] Where mean_angular is the average angular velocity; N is the number of IMU data frames received by the VIO system during the time between two adjacent frames of images arriving at the VIO system; and w is the angular velocity output by the gyroscope.

[0018] Preferably, converting the average angular velocity from the IMU coordinate system to the camera coordinate system and calculating the relative rotation matrix R_P_C comprises the following steps:

[0019] The average angular velocity is converted from the IMU coordinate system to the camera coordinate system according to formula (2):

[0020] cam_mean_angular=R_cam_imu*mean_angular(2)

[0021] Among them, R_cam_imu is the transformation matrix of the camera coordinate system relative to the IMU coordinate system;

[0022] According to formula (3), the relative rotation matrix R_P_C of two adjacent frames of images is obtained:

[0023] Rodrigues(cam_mean_angular*t,cam_R_p_c)(3)

[0024] Where t is the time difference between two adjacent frames of images arriving at the VIO system; cam_R_p_c represents the calculation result of cam_mean_angular*t expressed in Rodriguez form;

[0025] Perform matrix transposition of cam_R_p_c to obtain the R_P_C matrix.

[0026] Preferably, if the posture is unchanged and the current feature point of the first frame image is not in the image feature point set, the image feature point information of the current feature point is directly stored in the image feature point set.

[0027] Preferably, the current feature point A is obtained according to formula (4): k Corrected pixel coordinate pos_A k ':

[0028] pos_A k '=(H1*H2*H3*...H n-1 )*pos_A k (4)

[0029] Among them, H n-1 is the homography matrix of the adjacent n-1th frame image and the nth frame image, and H n-1 =K*R_P_C n-1 *K -1 ;R_P_C n-1 is the relative rotation matrix between the adjacent n-1th frame image and the nth frame image, and K is the camera intrinsic parameter.

[0030] Preferably, the unmanned equipment includes a drone.

[0031] Preferably, the situation where the posture remains unchanged includes the drone hovering in the air.

[0032] Preferably, it is determined whether the unmanned device is in a state of unchanged posture when acquiring n frames of images by the unmanned device's directional moving speed and its own posture change.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention can quickly complete the coordinate system transformation based on the average angular velocity to obtain the relative rotation matrix, and further continuously correct the feature point posture based on the relative rotation matrix to eliminate the influence of the unmanned equipment's own posture change on the feature point posture. Finally, based on the normal distribution principle and pixel coordinate difference, dynamic points are eliminated, thereby quickly completing the removal of dynamic feature points and retaining static points as much as possible to ensure the accuracy of subsequent posture calculation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Flowchart of the steps of the method for eliminating dynamic feature points based on normal distribution in the present invention;

[0036] Figure 2 Schematic diagram of the position change of the same feature point in n consecutive frames of images in the present invention;

[0037] Figure 3 is a histogram of pixel coordinate differences in the present invention;

[0038] Figure 4 To identify images with dynamic feature points that need to be removed and static feature points that need to be retained. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] Example 1:

[0041] like Figure 1 As shown, this embodiment provides a method for eliminating dynamic feature points based on normal distribution, which includes the following steps:

[0042] S1. Construct an image feature point set, and the set includes a plurality of image feature point information, and each image feature point information includes an image feature point index ID and the pixel coordinates posi of the image feature point in each frame of consecutive i frames, where i is a positive integer greater than or equal to 1, representing the number of consecutive frame images;

[0043] For example, in this embodiment, the image feature point set may be a Map data type, which stores index IDs and pixel coordinates as keys and values, respectively. Therefore, its data may be expressed in the following form: <feature point ID1: pos1, pos2, pos3, ..., posi>, <feature point ID2: pos1, pos2, pos3, ..., posi>, ..., <feature point IDN: pos1, pos2, pos3, ..., posi>, etc.; wherein ID1, ID2, ..., IDN are image feature point index IDs, and pos1, pos2, pos3, ..., posi represent the pixel coordinates of the image feature point in the consecutive 1st frame, 2nd frame, 3rd frame, ..., i-th frame, respectively;

[0044] In addition, the consecutive i-frame images are images of the same scene P and can be acquired by an imaging device mounted on an unmanned device while the posture of the unmanned device remains unchanged. The unmanned device includes a drone, and the imaging device includes a visible light and / or infrared camera. The unchanged posture includes the drone hovering in the air. At the same time, the image feature points can be extracted using a commonly used feature point extraction algorithm such as Sift / Fast / ORB, and the extracted data includes the pixel coordinates of the image feature points.

[0045] S2, receiving m consecutive frames of images of the same scene P and the IMU data of the unmanned device in real time through the VIO system of the unmanned device, and storing them in the image buffer and IMU data buffer respectively;

[0046] The average angular velocity is obtained based on the IMU data received by the VIO system during the time period between two adjacent frames of images arriving at the VIO system.

[0047] Since the IMU frame rate (e.g., 100 Hz) is much larger than the image frame rate (e.g., 30 Hz), when two adjacent frames of images arrive at the VIO system at time t1 and t2 respectively, the VIO system is also receiving IMU data (e.g., 3 frames of IMU data) during the time period between time t1 and t2. Therefore, the average angular velocity can be calculated for the IMU data received by the VIO system during the time period between time t1 and t2.

[0048] Furthermore, the average angular velocity is obtained according to formula (1):

[0049]

[0050] Wherein, mean_angular is the average angular velocity; N is the number of IMU data frames received by the VIO system during the time period between two adjacent frames of images arriving at the VIO system; w is the angular velocity output by the gyroscope, which is included in the IMU data and is expressed in radians per second. Specifically, in this embodiment, the average angular velocity mean_angular is a 3*1 vector, representing the average angular velocity of the gyroscope in the x, y, and z directions during the time period between two adjacent frames of images arriving at the VIO system.

[0051] S3. Convert the average angular velocity from the IMU coordinate system to the camera coordinate system and calculate the relative rotation matrix R_P_C, which specifically includes the following steps:

[0052] The average angular velocity is converted from the IMU coordinate system to the camera coordinate system according to formula (2):

[0053] cam_mean_angular=R_cam_imu*mean_angular(2)

[0054] Among them, R_cam_imu is the transformation matrix of the camera coordinate system relative to the IMU coordinate system, and the transformation matrix is a 3*3 matrix, which can be determined by a pre-calibration program;

[0055] According to the Rodrigo formula (i.e., formula (3)), the relative rotation matrix R_P_C of two adjacent frames of images is obtained:

[0056] Rodrigues(cam_mean_angular*t,cam_R_p_c)(3)

[0057] Where t is the time difference between two adjacent frames of images arriving at the VIO system, in seconds; cam_R_p_c represents the calculation result of cam_mean_angular*t expressed in Rodriguez form;

[0058] Perform matrix transposition of cam_R_p_c to obtain the R_P_C matrix, that is, the R_P_C matrix is the transposed matrix of cam_R_p_c;

[0059] Since the relative rotation matrix R_P_C represents the relative rotation relationship between two adjacent frames, the matrix can be used to predict the position of the feature points of the previous frame on the next frame.

[0060] S4. Obtain image feature point information of each feature point in the first frame of the image among the consecutive n frames of the image; the image feature point information of the feature point includes the index ID of the feature point A and the pixel coordinate pos_A of the feature point A in the first frame of the image; n is a positive integer greater than or equal to 2;

[0061] And judge whether the unmanned device is in a state of unchanged posture (such as hovering) when acquiring n frames of images by the unmanned device's directional movement speed and its own posture change; if so, execute step S5; if not, clear the map and execute step S7;

[0062] S5, determine whether the image feature point information of each feature point of the first frame image is in the image feature point set, if the current feature point A k In the image feature point set, the current feature point A is rotated based on the relative rotation matrix R_P_C. k Correct the pose to obtain the current feature point A k Corrected pixel coordinates pos_A in each frame except the first frame k ', and at the same time, the current feature point A in each frame image except the first frame image in the image feature point set k Pixel coordinate pos_A k Replaced with the corresponding corrected pixel coordinate pos_A k '; and each feature point in the first frame image appears in each frame image of the consecutive n frames;

[0063] If the current feature point A k If it is not in the image feature point set, then the current feature point A is directly k The image feature point information is stored in the image feature point set;

[0064] Furthermore, according to formula (4), the current feature point A is obtained k Corrected pixel coordinate pos_A k ':

[0065] pos_A k '=(H1*H2*H3*...H n-1 )*pos_A k (4)

[0066] Among them, H n-1 is the homography matrix of the adjacent n-1th frame image and the nth frame image, and H n-1 =K*R_P_C n-1 *K -1 ;R_P_C n-1 is the relative rotation matrix between the adjacent n-1th frame image and the nth frame image, K is the camera intrinsic parameter, which can be obtained through pre-calibration;

[0067] like Figure 2 As shown in the figure, when the posture of the unmanned equipment does not change, a feature point is A in the first frame image. 0However, due to the shaking and jittering of the unmanned equipment, the posture of the unmanned equipment changes, causing the feature point to appear in A in the nth frame image. N But at this time A N The feature points at (i.e., dynamic feature points) have obvious posture changes, so when they are used for subsequent posture calculations, the posture calculation results will be inaccurate;

[0068] As mentioned above, since the relative rotation matrix R_P_C can be used to predict the position and posture of the feature points in the next frame of image, the relative rotation matrix R_P_C is used in this embodiment to continuously correct the position and posture of the feature points to eliminate the influence of the position and posture change of the unmanned equipment itself on the position and posture of the feature points, thereby converting the A in the nth frame image into N The feature point pose at is corrected to A 0’ At this point, the pose of the feature point is constantly approaching the pose of the feature point in the image when the pose of the unmanned equipment does not change, that is, A 0 Therefore, A 0’ When the feature points at are used for subsequent pose calculations, the accuracy of the settlement results can also be guaranteed;

[0069] For example, assuming there are 5 consecutive frames of images, feature point A1 appears in each of the 5 frames of images, and in the image feature point set, the pixel coordinates of feature point A1 in the 5 frames of images are pos_A1, pos_A2, pos_A3, pos_A4, and pos_A5 respectively. Then, according to the homography matrix H1 of the 1st and 2nd frames of images, the corrected pixel coordinate position pos_A2' of the feature point A1 in the 2nd frame of image is predicted, and pos_A2'=H1*pos_A1. Similarly, according to the homography matrix H2 of the 2nd and 3rd frames of image, the corrected pixel coordinate position pos_A3' of the feature point A1 in the 3rd frame of image is predicted, and pos_A3'=H1*H2*pos _A1, and so on, until the corrected pixel coordinate pose pos_A5' of feature point A in the 5th frame image is obtained, and pos_A5'=H1*H2*H3*H4*pos_A1, so as to eliminate the deviation of the feature point pose in the consecutive frame images caused by the change of the unmanned equipment pose; further, the corrected pixel coordinate poses pos_A2', pos_A3', pos_A4', pos_A5' are replaced with the pixel coordinates pos_A2, pos_A3, pos_A4, pos_A5 respectively, so that in the image feature point set, the pixel coordinates of feature point A1 in the 5 frames of image are pos_A1, pos_A2', pos_A3', pos_A4', pos_A5' respectively;

[0070] S6. Repeat steps S4-S5. If the unmanned device does not change its posture when acquiring M consecutive frames of images, obtain the pixel coordinate difference of each feature point in the first frame and the M-th frame based on the pixel coordinates of the feature points stored in the current image feature point set, calculate the mean and standard deviation of all pixel coordinate differences, and eliminate feature points with pixel coordinate differences greater than three times the standard deviation.

[0071] The 3σ principle of normal distribution means that in a normal distribution, approximately 68% of the data values will be distributed within the interval of 1 standard deviation from the mean (i.e., the interval (μ-σ, μ+σ)), approximately 95% of the data values will be distributed within the interval of 2 standard deviations from the mean (i.e., the interval (μ-2σ, μ+2σ)), and approximately 99.73% of the data values will be distributed within the interval of 3 standard deviations from the mean (i.e., the interval (μ-3σ, μ+3σ));

[0072] In this embodiment, 5 consecutive frames of images are taken, and the pixel coordinate differences of 500 feature points in the images are statistically analyzed to generate the following: Figure 3 From the histogram shown, it is found that the pixel coordinate difference of each feature point in the first frame image and the fifth frame image conforms to the normal distribution characteristics. Since the pixel coordinate difference of most feature points will fall within the range of μ±3σ, the data points outside this range are considered to be abnormal dynamic feature points and are eliminated.

[0073] And S7, using all feature points in the current image feature point set for posture calculation of the unmanned device, and marking the feature points to be removed in the image.

[0074] like Figure 4 As shown in the figure, the red feature points in the image are the dynamic feature points that are removed and will not enter the subsequent aircraft attitude calculation process. The blue feature points are the static feature points that need to be retained and used for subsequent attitude calculation.

[0075] To sum up, the present application can obtain the position of the feature point based on IMU data to improve the subsequent tracking accuracy of the feature point and reduce the impact of the aircraft attitude change on the distance calculation. At the same time, the coordinate system conversion can be quickly completed through the average angular velocity to obtain the relative rotation matrix, and the feature point posture is further continuously corrected based on the relative rotation matrix to eliminate the impact of the unmanned equipment's own posture change on the feature point posture. Finally, the dynamic points are eliminated based on the normal distribution principle and the pixel coordinate difference. This can quickly complete the removal of dynamic feature points and retain static points as much as possible to ensure the accuracy of subsequent attitude calculation results.

[0076] It should be noted that, in this document, terms such as "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0077] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for eliminating dynamic feature points based on normal distribution, characterized in that: The steps include: Constructing an image feature point set, wherein the set includes a plurality of image feature point information, and each image feature point information includes an image feature point index ID and the pixel coordinates of the image feature point in each frame of consecutive i frames; Get the relative rotation matrix R_P_C between two adjacent frames of image; Obtain image feature point information for each feature point in the first frame of n consecutive image frames, and determine whether the unmanned device is in a state of unchanged posture when acquiring n frames of image frames; if it is in a state of unchanged posture, and the current feature point of the first frame of image is in the image feature point set, obtain the corrected pixel coordinates of the current feature point in each frame of image except the first frame of image based on the relative rotation matrix R_P_C, and replace the pixel coordinates of the current feature point in each frame of image except the first frame of image in the image feature point set with the corresponding corrected pixel coordinates; Also, when the unmanned equipment obtains M consecutive frames of images without any posture change, the pixel coordinate difference of each feature point in the 1st frame image and the Mth frame image is obtained according to the pixel coordinates of the feature points stored in the current image feature point set, and the mean and standard deviation of all pixel coordinate differences are calculated, and feature points with pixel coordinate differences greater than three times the standard deviation are eliminated.

2. The method for removing dynamic feature points according to claim 1, wherein: The image feature point set is of Map data type, which stores index ID and pixel coordinates as keys and values respectively.

3. The method for removing dynamic feature points according to claim 1, wherein: Obtaining the relative rotation matrix R_P_C between two adjacent frames of image includes the following steps: Receive m consecutive frames of images and IMU data through the VIO system, and obtain the average angular velocity based on the IMU data received by the VIO system during the time period between two adjacent frames of images arriving at the VIO system; The average angular velocity is converted from the IMU coordinate system to the camera coordinate system, and the relative rotation matrix R_P_C is calculated.

4. The method for removing dynamic feature points according to claim 3, wherein: The average angular velocity is obtained according to formula (1): Where mean_angular is the average angular velocity; N is the number of IMU data frames received by the VIO system during the time between two adjacent frames of images arriving at the VIO system; and w is the angular velocity output by the gyroscope.

5. The method for removing dynamic feature points according to claim 3, wherein: Convert the average angular velocity from the IMU coordinate system to the camera coordinate system and calculate the relative rotation matrix R_P_C, including the following steps: The average angular velocity is converted from the IMU coordinate system to the camera coordinate system according to formula (2): cam_mean_angular=R_cam_imu*mean_angular (2) Among them, R_cam_imu is the transformation matrix of the camera coordinate system relative to the IMU coordinate system; According to formula (3), the relative rotation matrix R_P_C of two adjacent frames of images is obtained: Rodrigues(cam_mean_angular*t,cam_R_p_c) (3) Where t is the time difference between two adjacent frames of images arriving at the VIO system; cam_R_p_c represents the calculation result of cam_mean_angular*t expressed in Rodriguez form; Perform matrix transposition of cam_R_p_c to obtain the R_P_C matrix.

6. The method for removing dynamic feature points according to claim 1, wherein: If the posture is unchanged and the current feature point of the first frame image is not in the image feature point set, the image feature point information of the current feature point is directly stored in the image feature point set.

7. The method for removing dynamic feature points according to claim 1, wherein: According to formula (4), the current feature point A is obtained k Corrected pixel coordinate pos_A k ': pos_A k ’=(H1*H2*H3*...H n-1 )*pos_A k (4) Among them, H n-1 is the homography matrix of the adjacent n-1th frame image and the nth frame image, and H n-1 =K*R_P_C n-1 *K -1 ;R_P_C n-1 is the relative rotation matrix between the adjacent n-1th frame image and the nth frame image, and K is the camera intrinsic parameter.

8. The method for removing dynamic feature points according to claim 1, wherein: The unmanned equipment includes a drone.

9. The method for removing dynamic feature points according to claim 1, wherein: The situation where the posture remains unchanged includes the drone hovering in the air.

10. The method for removing dynamic feature points according to claim 1, wherein: The unmanned device is judged by its directional moving speed and its own posture change whether it is in a state of unchanged posture when acquiring n frames of images.

Images