A binocular vision ranging method and system based on angular rotation consistency

Abstract

The present invention discloses a binocular vision ranging method and system based on angular rotation consistency, belonging to the field of vision ranging. Aiming at the problems of slow processing efficiency and low accuracy in existing vision ranging, the present invention provides a binocular vision ranging method based on angular rotation consistency, comprising the following steps: collecting two images and detecting a target to be measured; extracting feature points; performing feature point matching and removing mismatches; calculating the average disparity and the distance of the target to be measured. By calculating the rotation angles of paired feature points matched in two images, the present invention uses the consistency of feature point rotation to remove mismatched feature points, without the need to calculate a complex homography matrix, nor to perform repeated iterations. Only the rotation angles between paired feature points need to be calculated once, the rotation angles are divided into intervals, and removal is performed according to the quantity within the intervals, greatly improving the speed of screening mismatched feature points and enhancing real-time performance while ensuring accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual ranging, and more specifically, relates to a binocular visual ranging method and system based on angle rotation consistency. Background Art

[0002] Visual ranging has received extensive attention as one of the basic technologies in the field of machine vision. It occupies an important position in the field of robotics and is widely used in machine vision positioning, target tracking, visual obstacle avoidance, etc. Visual ranging is mainly divided into monocular ranging, binocular ranging, structured light ranging, etc. Among them, according to the binocular ranging model, it can be known that the distance from the object to be measured to the camera is inversely proportional to the disparity of the object observed by the binocular camera. Disparity refers to the sum of the distance between the object on the left camera imaging plane and the camera optical center and its distance from the optical center on the right camera imaging plane. The difficulty of binocular ranging lies in how to accurately calculate the disparity. Further speaking, it is how to accurately match the feature points in the two images, and calculate the disparity through the matched feature points. The accuracy of the matching will affect the measurement accuracy and efficiency. After the initial feature point matching, the existing method uses the RANSAC algorithm to remove the mismatched points to improve the ranging accuracy. However, in this way, the RANSAC algorithm is used to eliminate the mismatched points. The algorithm first selects 4 pairs of matching points to calculate the corresponding homography matrix H, and uses the matrix H to calculate the reprojection error of the remaining point pairs. If the error of the point pair is less than the threshold, the point pair is classified as an inlier belonging to the homography matrix, otherwise it is classified as an outlier. Repeat the above steps until the iteration upper limit is reached, and finally select the best homography matrix to remove the feature point pairs that do not conform to the homography matrix. This method has a large computational amount and poor real-time performance when there are many feature points.

[0003] Corresponding improvements have also been made to the above problems. For example, Chinese Patent Application No. CN201810029827.3, publication date July 6, 2018, this patent discloses a binocular stereo vision ranging method and its ranging device. The ranging method includes: calculating the rotation angle from the initial coordinate system to the standard coordinate system: setting an initial coordinate system in the binocular vision ranging system in each binocular vision ranging device. The initial coordinate system is referenced by two image sensors in the binocular vision ranging device, and the position coordinates of the two image sensors on the initial coordinate system will not change due to the change of the position of the corresponding binocular vision ranging device; determining the position coordinates one of all binocular vision ranging devices in the standard coordinate system according to the rotation angle; calculating the position coordinates two of the measured target in the initial coordinate system according to the difference obtained by measuring the same measured target by the two image sensors of the same binocular vision ranging device; calculating the position coordinates three of the measured target in the standard coordinate system according to the rotation angle, position coordinates one, and position coordinates two. The disadvantage of this patent is that although it can coordinate multiple devices to work simultaneously, the overall accuracy is general.

[0004] Another example is Chinese patent application number CN202011405478.4, which was published on February 26, 2021. The patent discloses a binocular ranging method for a vehicle blind spot detection alarm device, including: calibrating the parameters of the binocular camera; measuring the specific value of the rotation angle of the binocular camera relative to the side of the vehicle body; inputting the image collected by the binocular camera into the target detection algorithm, and the target detection algorithm outputs the target and determines the position information of the target in the image; calling the binocular stereo matching algorithm to determine the parallax of the target in the two frames of the image; based on the focal length, parallax, and the distance between the lenses, determine the first distance from the projection point of the target on the plane where the line between the optical axis and the lens is located to the vertical point of the line between the lenses; based on the rotation angle and roll angle, the coordinates of the target in the image coordinate system, the coordinates of the optical center, the focal length and the first distance of the target, the vertical distance from the target to the vehicle body is obtained; the vertical distance from the target to the vehicle is the shortest distance between the target and the vehicle. The shortcomings of this patent are: complex operation, slow overall efficiency, and poor real-time performance. Summary of the invention

[0005] 1. Problems to be solved

[0006] In view of the problems of slow processing efficiency and low accuracy of existing visual ranging, the present invention provides a binocular visual ranging method and system based on angle rotation consistency. The method of the present invention calculates the rotation angle of matched pairs of feature points in two images, and uses the consistency of feature point rotation to eliminate mismatched feature points. It does not need to calculate a complex homography matrix, nor does it need to be repeatedly iterated. It only needs to calculate the rotation angle between pairs of feature points once, divide the interval according to the rotation angle, and then eliminate the mismatched feature points, thereby ensuring the accuracy while enhancing the overall real-time performance. The system structure of the present invention is simple, and the system working stability is ensured while improving the degree of automation, and the efficiency is high.

[0007] 2. Technical solution

[0008] To solve the above problems, the present invention adopts the following technical solutions.

[0009] A binocular vision ranging method based on angle rotation consistency comprises the following steps:

[0010] S1: Use a binocular camera to capture two images and detect the position of the target in the image respectively;

[0011] S2: Extract feature points of the targets detected in the two images respectively;

[0012] S3: Match the feature points in the two images;

[0013] S4: Calculate the rotation angle between two matching feature points, classify a number of rotation angles into intervals, and record the two matching feature points corresponding to the rotation angle within the interval. Select the three intervals with the top three numbers of feature points to obtain all the matching feature points within the three intervals. The interval classification is to increment from 0° to 360° in units of scale m°, forming intervals;

[0014] S5: Calculate the average parallax d of all the matching feature points within the three intervals:

[0015]

[0016] where: u pi is the abscissa of the feature point in one image; u qi is the abscissa of the feature point in the other image that matches u pi ; n is the number of pairs of matching feature points; d x is the scale of the pixel;

[0017] S6: Calculate the distance from the target to be measured to the binocular camera according to R, where f is the focal length of the binocular camera, b is the baseline of the binocular camera, and d is the average parallax.

[0018] Furthermore, in step S2, the FAST corner points are selected as the feature points of the target to be measured, and at the same time, the direction and descriptor of the FAST corner points are calculated.

[0019] Furthermore, step S3 specifically includes the following steps:

[0020] S31: Select a feature point in one image;

[0021] S32: Calculate the Hamming distance between the feature point in step S31 and several feature points in the other image, and select the feature point with the smallest Hamming distance from the feature point in step S31 in the other image as the matching point;

[0022] S33: Repeat steps S31 and S32 to complete the matching of all feature points in the two images.

[0023] Furthermore, step S4 specifically includes the following steps:

[0024] S41: Divide the interval into 30 intervals at a unit scale of 12°. The first interval is 0 - 12°, the second interval is 12 - 24°, and so on, for a total of 30 intervals;

[0025] S42: Calculate the rotation angle between two mutually matching feature points in two images, where the rotation angle is the absolute value of the difference between the direction angles of the two feature points;

[0026] S43: Divide intervals according to the rotation angle. After determining the intervals, record the two feature points corresponding to the rotation angle within the intervals;

[0027] S44: Repeat steps S42 and S43 until all feature points are recorded in the corresponding intervals;

[0028] S45: Screen out the three intervals with the top three numbers of feature points.

[0029] Furthermore, after step S3, further screening of the three intervals is also included, specifically: calculate the average rotation angles of all feature points within the three intervals respectively. The average rotation angle of the interval with the largest number of interval feature points is subtracted from the average rotation angles of the other two intervals respectively. If the absolute value of the difference is less than the threshold, then the other two intervals are retained; if it is greater than the threshold, then the other two intervals are discarded.

[0030] Furthermore, in step S1, the SSD algorithm is used to detect the position of the object to be measured in the image.

[0031] A system using the binocular vision ranging method based on angular rotation consistency as described in any one of the above, comprising:

[0032] Acquisition module: used to take images of the object to be measured;

[0033] Detection module: used to detect the object to be measured in the images in the acquisition module;

[0034] Feature point extraction module: used to extract feature points of the object to be measured in the images in the detection module;

[0035] Matching module: used to match the feature points in two images in the feature point extraction module one by one;

[0036] Rejection of mis-matched point module: used to screen the paired feature points after matching and reject the mis-matched feature points;

[0037] Average disparity calculation module: used to calculate the average disparity of all the remaining paired feature points;

[0038] Distance calculation module: used to calculate the distance of the object to be measured.

[0039] Furthermore, it further includes an alarm module: used to monitor and give early warnings to each module in real time.

[0040] 3. Beneficial effects

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0042] (1) By calculating the rotation angles of the matching paired feature points in two images, the present invention uses the consistency of the feature point rotation to eliminate the mismatched feature points. It does not require calculating a complex homography matrix, nor does it need to perform repeated iterations. Only the rotation angles between the paired feature points need to be calculated once, the rotation angle range is divided, the number of paired feature points in several intervals is counted in sequence, and the top three intervals with the largest number are retained for subsequent distance measurement calculation, while the others are eliminated as mismatched points. The whole method greatly improves the speed of screening mismatched feature points, enhances the overall real-time performance while ensuring the accuracy, is easy to operate, and is suitable for occasions with high real-time requirements;

[0043] (2) When extracting feature points in the image, the present invention uses FAST corner points as feature points. Since FAST corner points are points with a large brightness difference from the surrounding pixels in the image, using them as feature points further ensures the accuracy and provides an accurate data source for the subsequent distance measurement results; and it can simply and conveniently calculate the corresponding feature point directions and descriptors, making the subsequent feature point matching and rotation angle calculation faster, further improving the work efficiency; when performing feature point matching, the Hamming distance is used to match between two feature points, effectively reducing the matching error and ensuring the accuracy;

[0044] (3) After screening out three intervals, the present invention further screens the three intervals. By calculating the average rotation angles in the three intervals respectively, the difference is made between the average angle of the interval with the largest number of feature point pairs and the average angles of the other two remaining intervals, and the absolute value of the result is compared with the threshold to eliminate or retain the paired feature points in the interval. Further eliminating or retaining the data on the basis of accurate results further increases the accuracy of eliminating mismatched feature points, reduces the workload for subsequent calculation work, and improves the calculation speed while ensuring the accuracy of the results;

[0045] (4) The system of the present invention corresponds to each module to execute corresponding functions, has a simple structure, and the work of each other does not interfere with each other, improving the automation degree while ensuring the working stability of the system, reducing the input of labor costs, and then improving the work efficiency; at the same time, the alarm module monitors and warns each module in real time, facilitating the staff to discover problems and make adjustments in time, improving the safety and timeliness of the whole system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a schematic flow chart of the present invention;

[0047] Figure 2This is a schematic flowchart for eliminating mis-matched points in the present invention. Detailed implementation manners

[0048] The present invention will be further described below in conjunction with specific embodiments and the accompanying drawings.

[0049] The core idea of this application is to eliminate mis-matched points through the rotational consistency of feature points. Here, the rotation angle is the difference in the direction angles of the two matched feature points. Since there is only translation and no rotation between the two images obtained by the binocular camera, ideally, the difference in the rotation angles between all the two matched feature points should be zero. However, due to the existence of errors such as noise or other external environments, the difference in the rotation angles between the two actually matched feature points is not zero. Therefore, this application mainly uses the consistency of feature point rotation to eliminate feature points, so as to achieve the purpose of high accuracy and high real-time performance. A detailed description will be given next.

[0050] Embodiment 1

[0051] As Figure 1 and Figure 2 shown, a binocular vision ranging method based on angular rotation consistency includes the following steps:

[0052] S1: Use a binocular camera to photograph the target to be measured, thereby collecting two images, and respectively detect the positions of the target to be measured in the two images; in this step, it is defined that since the binocular acquisition can collect two images at the same time, according to the position, one image is called the left image and the other image is called the right image; and the SSD algorithm is selected to respectively detect the positions of the target to be measured in the left and right images. This algorithm has high detection accuracy and high efficiency;

[0053] S2: Extract feature points from the targets to be measured detected in the two images respectively; specifically, in this step, the FAST corner points are selected as the feature points of the target to be measured. Since the FAST corner points are points in the image with a large brightness difference from the surrounding pixels, it only needs to compare the brightness of the pixel point to be extracted with the brightness of 16 pixels on the circle with it as the center and a radius of 3. If the brightness difference from more than 9 of these pixels exceeds 20%, then the pixel point to be extracted is considered a FAST corner point; using the FAST corner points as feature points further ensures the accuracy rate and provides accurate data sources for the subsequent ranging results;

[0054] Calculate the direction and descriptor of FAST corner points simultaneously. The calculation of the direction of FAST corner points is as follows: Taking the FAST corner point as the center, select a window with a radius of r pixels. Here, r is empirically set to 31. According to the selected window, calculate the centroid of the image patch. After calculating the centroid, select the direction from the FAST corner point to the centroid as the direction of the FAST corner point. The descriptor of the FAST corner point is as follows: The descriptor used in the algorithm consists of 256-bit binary numbers, and the descriptor represents the magnitude relationship of the gray values of 256 pairs of pixel points near the FAST corner point.

[0055] S3: Match the feature points in the two images; the so-called matching means that the feature points in the left image match the feature points in the right image to form a pair of matching feature points; specifically, it includes the following steps:

[0056] S31: Select a feature point in the left image;

[0057] S32: Calculate the Hamming distance between this feature point in the left image and several possible matching feature points in the right image. The Hamming distance is the number of different bits between the descriptors of the feature point in the left image and the feature point in the right image; Select the feature point in the right image with the smallest Hamming distance from the feature point in the left image as the matching point, and match it with the feature point in the left image to obtain a pair of matching feature points;

[0058] S33: Repeat steps S31 and S32 to complete the matching of all feature points in the two images.

[0059] In this step, the feature point with the smallest Hamming distance is used as the matching point for the feature point in the left image, which fully compares the feature points in the two images in all aspects, effectively reduces the matching error, and then ensures the matching accuracy, provides stable and accurate data for subsequent calculations, and ensures a relatively high accuracy of the final measurement result.

[0060] S4: Calculate the rotation angle between the two matching feature points, classify several rotation angles into intervals, and record the two matching feature points corresponding to the rotation angle within the interval. Select the intervals with the top three numbers of feature points to obtain all the matching feature points within the three intervals; The interval classification is to increment from 0° to 360° according to the unit scale m°, forming intervals. The value of m is generally determined according to the actual situation on site and is not specifically limited here. It can be defined by itself according to different situations; In this embodiment, through experiments and theories, m is set to 12. This step specifically includes the following steps:

[0061] S41: Divide the interval into 30 intervals with a unit scale of 12°, the first interval is 0-12°, the second interval is 12-24°, and so on, increasing by 12 in sequence, to divide into 30 intervals;

[0062] S42: Calculate the rotation angle between two feature points that match each other in the two images, where the rotation angle is the absolute value of the difference between the direction angles of the two feature points; the direction angle refers to the direction in which the feature point points to its corresponding centroid. For example, there is a feature point p in the left image and a feature point q that matches p in the right image. The direction angle of p is β, and the direction angle of q is γ. Then the rotation angle of the pair of matching feature points is |β-γ|.

[0063] S43: Divide the interval according to the rotation angle, and after determining the interval, record the two feature points corresponding to the rotation angle in the interval; for example: the sixth pair of feature points p in the left and right images 6 and q 6 , the calculated rotation angle is 30°, which corresponds to the third interval. The number of feature point pairs in the interval is added by 1, that is, the interval is counted, and the index of the feature point pair is saved, that is, the index is saved as 6; as long as the rotation angle falls into the interval, the interval is counted by 1 successively, and the two feature points corresponding to the rotation angle are recorded in the interval;

[0064] S44: repeating step S42 and step S43 until all feature points (i.e., all feature points that are paired and matched with each other) are recorded in the corresponding interval;

[0065] S45: Filter out the three intervals with the first three feature points, and remove the feature points in the remaining intervals without retaining them. The three intervals are selected to prevent the omission of some correctly matched feature points, thereby reducing the workload and improving work efficiency while ensuring the accuracy.

[0066] S5: Calculate the average disparity d of all matching feature points in the three intervals, make a difference in the horizontal coordinates of the pixel coordinates of all retained pairs of matching feature points in the image, then sum up the absolute values ​​of all the differences, calculate the average value dm, and finally multiply it by the pixel scale d x Get the average disparity, where d x Represents the actual length of each pixel; for example, there are n pairs of matching feature points Where p is the feature point on the left image, q is the feature point on the right image, p and q match each other, the specific formula is as follows:

[0067]

[0068] Where: u pi u is the horizontal coordinate of the pixel coordinate of a feature point in an image;qi is the abscissa of the pixel coordinates of the feature point matching u in another image; n is the number of pairs of matching feature points; d pi is the scale of the pixel; x

[0069] S6: According to calculate the distance from the target to be measured to the binocular camera, where f is the focal length of the binocular camera, b is the baseline of the binocular camera, and d is the average parallax.

[0070] The method of the present invention utilizes the characteristic that the rotation angles between two images are consistent. By statistically analyzing the rotation angles between mutually matching feature points, the mismatched feature points are eliminated. It does not require calculating a complex homography matrix, nor does it require repeated iterations. Only the rotation angles between pairs of feature points need to be calculated once. The rotation angles are divided into intervals, and the number of pairs of feature points in several intervals is statistically analyzed in sequence. The first three intervals with the largest number are retained for subsequent distance measurement calculations, and the others are eliminated as mismatched points. The entire method greatly improves the speed of screening mismatched feature points, enhances the overall real-time performance while ensuring accuracy, is easy to operate, and is applicable to occasions with high real-time requirements, such as calculating the real-time distance of a car driving on the road.

[0071] Embodiment 2

[0072] Basically the same as Embodiment 1, after step S3, it further includes further screening of the three intervals, specifically:

[0073] Calculate the average rotation angles of all feature points in the three intervals respectively. The three intervals are A, B, and C, where the number of feature points in the three intervals is A > B > C. Calculate the average rotation angles m A , m B , m C ;

[0074] The average rotation angle of the interval with the largest number of feature points is subtracted from the average rotation angles of the other two intervals respectively. If the absolute value of the difference is less than the threshold, then the other two intervals are retained; if it is greater than the threshold, then the other two intervals are discarded. That is, set the threshold as ε, and the setting of the threshold can be determined according to specific circumstances. Generally speaking, usually the threshold ε is 20% of m A ; if |m A - m B | < ε, then all feature points in interval B are retained, otherwise all feature points in interval B are discarded; if |m A - m C | < ε, then all feature points in interval C are retained, otherwise all feature points in interval C are discarded.

[0075] ​Further eliminate or retain data based on accurate results, thereby further increasing the accuracy of eliminating mis-matched feature points, reducing the workload for subsequent calculation work, ensuring accurate results while improving the calculation speed.

[0076] Embodiment 3

[0077] A system using the binocular vision ranging method based on angular rotation consistency described in any one of the above Embodiments 1-2, comprising:

[0078] An acquisition module: used to take images of the object to be measured, and the imaging tool is a binocular camera;

[0079] A detection module: used to detect the target to be measured in the images in the acquisition module, that is, to detect the position of the target to be measured in the images;

[0080] A feature point extraction module: used to extract feature points of the target to be measured in the images in the detection module;

[0081] A matching module: used to match the feature points in the two images in the feature point extraction module one by one;

[0082] A mis-matched point elimination module: used to screen the paired feature points after matching and eliminate the mis-matched feature points; this module mainly calculates the rotation angle between the paired feature points, determines the interval to which they belong according to the rotation angle, records the paired feature points and the quantity within this interval, and then performs interval screening to select the three intervals with the top three feature point quantities;

[0083] An average parallax calculation module: used to calculate the average parallax of all paired feature points within the remaining three intervals;

[0084] A distance calculation module: used to calculate the distance of the object to be measured;

[0085] It also includes an alarm module: used to monitor and give early warnings to each module in real time.

[0086] The system of the present invention performs corresponding functions through each module, has a simple structure, and the work of each other does not interfere with each other, ensuring the working stability of the system while improving the degree of automation, reducing the input of labor costs, and then improving work efficiency; at the same time, through the alarm module, each module is monitored and warned in real time, which is convenient for the staff to discover problems in time and make adjustments, improving the safety and timeliness of the entire system.

[0087] The examples described in the present invention are only descriptions of the preferred embodiments of the present invention, and do not limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various deformations and improvements made by those skilled in the art to the technical solutions of the present invention shall fall within the protection scope of the present invention.

Claims

1. A binocular vision ranging method based on angle rotation consistency, Features: The following steps are involved: S1: Use a binocular camera to capture two images and detect the position of the target in the image respectively; S2: Extract feature points of the targets detected in the two images respectively; S3: Match the feature points in the two images; S4: Calculate the rotation angle between two matching feature points, classify several rotation angles into intervals, and correspondingly record the two matching feature points corresponding to the rotation angle within the interval. Select the top three intervals in terms of the number of feature points to obtain all the matching feature points within the three intervals; the interval classification is to increment from 0° to 360° in accordance with the unit scale of m°, forming intervals; S5: Calculate the average disparity d of all matching feature points in the three intervals: Where: u pi is the abscissa of a feature point in an image; u qi is the abscissa of the feature point in another image that matches u pi ; n is the number of pairs of matching feature points; d x is the scale of the pixel; S6: According to calculate the distance from the target to be measured to the binocular camera, where f is the focal length of the binocular camera, b is the baseline of the binocular camera, and d is the average parallax.

2. A binocular vision ranging method based on angle rotation consistency according to claim 1, Features: In the step S2, FAST corner points are selected as feature points of the target to be measured, and the directions of the FAST corner points and the descriptors of the FAST corner points are calculated at the same time.

3. A binocular vision ranging method based on angle rotation consistency according to claim 1 or 2, Features: Step S3 specifically includes the following steps: S31: Select a feature point in an image; S32: Calculate the Hamming distance between the feature point in step S31 and several feature points in another image, and select the feature point in another image with the smallest Hamming distance to the feature point in step S31 as the matching point; S33: Repeat steps S31 and S32 to complete the matching of all feature points in the two images.

4. The binocular vision ranging method based on angle rotation consistency according to claim 1, Features: The step S4 specifically includes the following steps: S41: Divide the interval into 30 intervals with a unit scale of 12°, the first interval is 0 to 12°, the second interval is 12 to 24°, and so on, to divide the interval into 30 intervals; S42: Calculate the rotation angle between two feature points that match each other in the two images, where the rotation angle is the absolute value of the difference between the direction angles of the two feature points; S43: Divide the interval according to the rotation angle, and after determining the interval, record two feature points corresponding to the rotation angle in the interval; S44: repeating step S42 and step S43 until all feature points are recorded in the corresponding interval; S45: Filter out three intervals with the top three feature point quantities.

5. A binocular vision ranging method based on angle rotation consistency according to claim 1 or 4, Features: After step S3, the three intervals are further screened, specifically: the average rotation angles of all feature points in the three intervals are calculated respectively, and the average rotation angle of the interval with the largest number of feature points is subtracted from the average rotation angles of the other two intervals. If the absolute value of the difference is less than a threshold, the other two intervals are retained; if it is greater than the threshold, the other two intervals are discarded.

6. The binocular vision ranging method based on angle rotation consistency according to claim 1, Features: In the step S1, the SSD algorithm is used to detect the position of the target to be detected in the image.

7. A system using the binocular visual ranging method based on angle rotation consistency as claimed in any one of claims 1 to 6, Features: include: Acquisition module: used to capture images of the object to be measured; Detection module: used to detect the target to be measured from the images in the acquisition module; Feature point extraction module: used to extract feature points of the target to be measured from the images in the detection module; Matching module: used to match the feature points in the two images in the feature point extraction module one by one; False matching point elimination module: used to screen the paired feature points after matching and eliminate the falsely matched feature points; Average parallax calculation module: used to calculate the average parallax of all the remaining paired feature points; Distance calculation module: used to calculate the distance of the object to be measured.

8. A binocular vision ranging system based on angular rotation consistency according to claim 7, wherein: It further includes an alarm module: used to monitor and give early warnings to each module in real time.

Images