Binocular vision two-dimensional-three-dimensional coordinate conversion method based on multi-feature fusion

Abstract

The invention provides a binocular vision two-dimensional-three-dimensional coordinate conversion method based on multi-feature fusion. The binocular vision two-dimensional-three-dimensional coordinate conversion method comprises the steps of S10, initialization setting, S20, ORB pre-screening, S30, SIFT / SURF accurate extraction, S40, dynamic weighted fusion, S50, feature point matching and optimization and S60, three-dimensional coordinate calculation. According to the invention, through a hierarchical cooperation and dynamic weighting multi-feature fusion method, a high-quality feature point set is output for binocular matching and three-dimensional coordinate conversion, and rapid and high-precision extraction and matching of feature points in a binocular vision image are realized, so that the precision, real-time performance and scene adaptability of two-dimensional-three-dimensional coordinate conversion are improved; the method solves the problem that a two-dimensional-three-dimensional coordinate conversion method in the prior art is difficult to consider precision and speed at the same time, and can be widely applied to the fields of three-dimensional reconstruction, automatic driving, robot navigation, industrial detection, virtual reality and the like.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a binocular vision two-dimensional to three-dimensional coordinate transformation method based on multi-feature fusion. Background Technology

[0002] In fields such as 3D reconstruction, autonomous driving, robot navigation, and industrial inspection, the principle of accurate conversion from 2D to 3D spatial coordinates is as follows: 2D images of the target scene are acquired by two cameras (left and right), feature points in the images are extracted and cross-view matching is completed, and then the camera intrinsic parameters (focal length, pixel size, etc.) and extrinsic parameters (baseline distance, relative pose, etc.) are combined to calculate the 3D coordinates of the feature points using the principle of triangulation, and finally a 3D model of the scene is constructed or the target is located.

[0003] Current technologies often employ single algorithms or simple combinations, such as using the ORB algorithm for feature point extraction followed by the SIFT algorithm for matching. These approaches fail to simultaneously meet the demands for high accuracy and real-time performance. Specifically, while a single SIFT / SURF algorithm offers high accuracy, its speed is insufficient; conversely, a single ORB algorithm, while fast, has limited accuracy. Simple combination schemes do not address the issue of complementary strengths between algorithms, resulting in low matching efficiency and poor robustness. Therefore, there is an urgent need for a feature point extraction and matching method that integrates the advantages of multiple algorithms and balances accuracy and speed to improve the overall performance of binocular vision 2D-3D coordinate transformation.

[0004] In view of this, the present invention is proposed. Summary of the Invention

[0005] To address the shortcomings of the aforementioned technologies, this invention provides a binocular vision 2D-3D coordinate transformation method based on multi-feature fusion. It improves extraction speed through ORB pre-screening to meet real-time requirements of over 20fps; it enhances matching accuracy through dynamic weighted fusion to meet high-precision requirements, reduces mismatch rate, and lowers 3D coordinate transformation error by 50% to 80% compared to existing solutions.

[0006] The specific details of the technical solution provided by this invention are as follows:

[0007] A binocular vision 2D-3D coordinate transformation method based on multi-feature fusion includes:

[0008] S10. Initialize settings: Set the binocular camera parameters, image resolution, ORB extraction parameters, SIFT / SURF extraction parameters, and fusion weight calculation threshold.

[0009] S20. For the input images from the left and right cameras in the binocular camera, the ORB algorithm is used to quickly extract candidate feature points and obtain the two-dimensional coordinates and preliminary rotation direction of high-quality candidate feature points.

[0010] S30. Simultaneously start the SIFT module and SURF module to accurately extract the high-quality candidate feature points obtained in step S20.

[0011] S40. Dynamically adjust the weights of SIFT and SURF based on the complexity of the scene texture to generate a fused feature descriptor;

[0012] S50. For the fused feature point set of the left and right cameras obtained in step S40, the same point pairing is completed by Euclidean distance matching and RANSAC mismatch removal, and the accurate matching result is output.

[0013] S60. Combining the intrinsic and extrinsic parameters of the binocular camera with the two-dimensional coordinates of the matched corresponding points, the three-dimensional coordinates of the feature points are calculated using the principle of triangulation, thus completing the two-dimensional to three-dimensional coordinate transformation.

[0014] Furthermore, the binocular camera parameters include intrinsic and extrinsic parameters, where the intrinsic parameters include focal length f and pixel size dx / dy, and the extrinsic parameters include baseline distance B and relative pose angle θ; the fusion weight calculation threshold includes texture complexity threshold T.

[0015] Furthermore, in step S20, the ORB algorithm is used to quickly extract candidate feature points, including:

[0016] By optimizing the candidate set through contrast filtering and edge culling, invalid points with low contrast and edge interference are eliminated, and the two-dimensional coordinates (u, v) and preliminary rotation direction of high-quality candidate feature points are output.

[0017] Furthermore, in step S30,

[0018] The SIFT module's precise extraction of high-quality candidate feature points includes: constructing a Gaussian difference pyramid, relocating the scale space extrema of candidate points, generating a 128-dimensional high-precision feature descriptor, and recording scale information and response intensity.

[0019] The SURF module's accurate extraction of high-quality candidate feature points includes: calculating the Hessian matrix response around the candidate point using the integral image, determining the optimal scale, generating a 64-dimensional robust feature descriptor, and recording the contrast and response intensity.

[0020] Furthermore, the parameters of the Gaussian difference pyramid include 6 scale levels and 5 octaves.

[0021] Furthermore, step S40 dynamically adjusts the weights of SIFT and SURF based on the scene texture complexity to generate a fused feature descriptor, specifically including:

[0022] S401. Texture complexity calculation: For each candidate feature point, calculate the pixel gradient variance σ² of its 3×3 neighborhood. If σ²≥T, it is determined as a high-texture area; if σ²<T, it is determined as a low-texture area.

[0023] S402. Weight assignment: The high-texture area is assigned a SIFT weight range of 0.6 - 0.7 and a SURF weight range of 0.3 - 0.4; the low-texture area is assigned a SURF weight range of 0.4 - 0.5 and a SIFT weight range of 0.5 - 0.6.

[0024] S403. Descriptor fusion: Expand the 64-dimensional descriptor of SURF to 128 dimensions by linear interpolation, and perform weighted summation with the 128-dimensional descriptor of SIFT according to the above weights to obtain a 128-dimensional fused descriptor.

[0025] Further, in step S50, the pairing of homologous points is completed by Euclidean distance matching and RANSAC outlier rejection, including:

[0026] S501. Perform L2 normalization preprocessing on the 128-dimensional fused descriptors generated by the left and right cameras to eliminate the influence of the difference in the numerical ranges of different feature point descriptors on matching.

[0027] S502. For the fused descriptor of each feature point of the left camera, calculate its Euclidean distance from the fused feature descriptors of all feature points of the right camera, and use the nearest neighbor ratio strategy to screen potential homologous points. <00​​​​​​​​​​​​​​​​​​​

[0034] Where d LK d RK These are the k-th dimension components of the left and right camera descriptors, respectively.

[0035] Furthermore, calculating the three-dimensional coordinates of feature points using the principle of triangulation includes:

[0036] By using a binocular camera to obtain the two-dimensional projected coordinates of the same feature point from different perspectives, and combining the camera's intrinsic and extrinsic parameters, the three-dimensional spatial coordinates of the feature point are inferred through geometric relationships.

[0037] This invention provides a binocular vision 2D-3D coordinate transformation method based on multi-feature fusion, which has the following beneficial technical effects compared with the prior art:

[0038] (1) By using ORB pre-screening to improve extraction speed, the overall time consumption is reduced to 30~50ms / image, which is 2~3 times faster than pure SIFT and meets the real-time requirement of more than 20fps; by using dynamic weighted fusion to improve matching accuracy, the high precision requirement is met, the mismatch rate is less than 2%, and the three-dimensional coordinate transformation error is reduced by 50%~80% compared with the existing technology.

[0039] (2) The scene adaptability is greatly enhanced. The weight can be dynamically adjusted according to the texture complexity to adapt to various scenes such as high texture (industrial parts), low texture (glass surface), lighting changes (indoor and outdoor switching), and target scale changes (dynamic vehicles);

[0040] By optimizing the quality of candidate points through contrast filtering and edge culling, and reducing interference from invalid feature points, the robustness in complex scenarios is further improved.

[0041] (3) ORB (open source) and SURF (open source version) are used as core modules to avoid SIFT patent risks and reduce commercial application costs; no need to rely on GPU acceleration, can be implemented on ordinary CPUs (such as Intel i5), reducing hardware costs and engineering implementation difficulty;

[0042] (4) It can be directly applied to fields such as 3D reconstruction, autonomous driving, robot navigation, industrial inspection, and virtual reality (VR), providing high-quality feature point support for 2D-3D coordinate transformation in various scenarios; it can be extended to multi-view vision systems, further improving the accuracy of 3D coordinate transformation through multi-camera feature point fusion. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0044] In the description of this invention, it should be noted that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "inner", "outer", etc., indicating orientation or positional relationships, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0045] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0046] This invention discloses a binocular vision 2D-3D coordinate transformation method based on multi-feature fusion, comprising:

[0047] S10. Initialize settings: Set the binocular camera parameters, image resolution, ORB extraction parameters, SIFT / SURF extraction parameters, and fusion weight calculation threshold.

[0048] S20. For the input images from the left and right cameras in the binocular camera, the ORB algorithm is used to quickly extract candidate feature points and obtain the two-dimensional coordinates and preliminary rotation direction of high-quality candidate feature points.

[0049] S30. Simultaneously start the SIFT module and SURF module to accurately extract the high-quality candidate feature points obtained in step S20.

[0050] S40. Dynamically adjust the weights of SIFT and SURF based on the complexity of the scene texture to generate a fused feature descriptor;

[0051] S50. For the fused feature point set of the left and right cameras obtained in step S40, the same point pairing is completed by Euclidean distance matching and RANSAC mismatch removal, and the accurate matching result is output.

[0052] S60. Combining the intrinsic and extrinsic parameters of the binocular camera with the two-dimensional coordinates of the matched corresponding points, the three-dimensional coordinates of the feature points are calculated using the principle of triangulation, thus completing the two-dimensional to three-dimensional coordinate transformation.

[0053] It should be noted that the speed problem is solved by ORB pre-filtering in step S20. ORB pre-filtering reduces the overall time to 30~50ms / image, which is 2~3 times faster than pure SIFT and meets the real-time requirement of more than 20fps.

[0054] By using SIFT / SURF precise extraction and dynamic weighted fusion in step S40 to improve matching accuracy, the matching accuracy rate reaches over 95%, the false matching rate is less than 2%, and the 3D coordinate transformation error is reduced by 50% to 80% compared with existing technologies.

[0055] Furthermore, dynamic weighted fusion can dynamically adjust the weights according to the texture complexity, adapting to various scenarios such as high texture (industrial parts), low texture (glass surface), lighting changes (indoor-outdoor switching), and target scale changes (dynamic vehicles), thus solving the problem of scene adaptation.

[0056] In some embodiments, the binocular camera parameters include intrinsic and extrinsic parameters, wherein the intrinsic parameters include focal length f and pixel size dx / dy, and the extrinsic parameters include baseline distance B and relative pose angle θ; the fusion weight calculation threshold includes texture complexity threshold T.

[0057] Furthermore, ORB extraction parameters include the initial window size and the number of extractions; SIFT / SURF extraction parameters include the number of scale layers and the descriptor dimension.

[0058] For example, the image resolution is 1920×1080; the fusion weight calculation threshold includes the texture complexity threshold T, T=50.

[0059] In some embodiments, step S20 employs the ORB algorithm to quickly extract candidate feature points, including:

[0060] By optimizing the candidate set through contrast filtering and edge culling, invalid points with low contrast and edge interference are eliminated, and the two-dimensional coordinates (u, v) and preliminary rotation direction of high-quality candidate feature points are output.

[0061] Specifically, the ORB algorithm for quickly extracting candidate feature points includes the following steps:

[0062] S201. Using the "neighborhood pixel difference judgment method", a certain pixel in the image is selected as a candidate point, and a neighborhood with a radius of 3 (containing 16 pixels) is set.

[0063] If there are 9 or more consecutive pixels in the neighborhood whose grayscale value difference with the candidate point exceeds a set threshold (the set threshold can be dynamically adjusted; for example, the set threshold can be set to 20), then it is determined to be a corner point. At the same time, the principal direction θ is calculated for each corner point using the grayscale centroid method to achieve rotation invariance.

[0064] S202. After optimization through "contrast filtering (calculating the gray-level variance of the 3×3 neighborhood of candidate points, and removing those less than 10) and edge removal (calculating the response value of the Harris matrix, and removing those less than 0.1)," the coordinates of high-quality candidate points and the initial rotation direction are output, where the initial rotation direction is the principal direction θ calculated by ORB.

[0065] The embodiments described above in this application optimize candidate point quality through contrast filtering and edge culling, reduce interference from invalid feature points, and improve robustness in complex scenarios.

[0066] In some embodiments, in step S30, the SIFT module and the SURF module are simultaneously started to accurately extract the high-quality candidate feature points obtained in step S20, as follows:

[0067] The SIFT module's precise extraction of high-quality candidate feature points includes: constructing a Gaussian difference pyramid, relocating the scale space extrema of candidate points, generating a 128-dimensional high-precision feature descriptor, and recording scale information and response intensity.

[0068] The SURF module's accurate extraction of high-quality candidate feature points includes: calculating the Hessian matrix response around the candidate point using the integral image, determining the optimal scale, generating a 64-dimensional robust feature descriptor, and recording the contrast and response intensity.

[0069] It should be further explained that, based on the high-quality candidate points obtained in step S20, a 128-dimensional descriptor is generated by constructing a Gaussian difference pyramid using SIFT and a 64-dimensional descriptor is generated by using SURF with integral images. The candidate point coordinates output by ORB can reduce the SIFT scale space search range to an 11×11 pixel area centered on the coordinates. The initial rotation direction passed can allow SIFT to statistically analyze the gradient direction within a range of ±15°, reducing the amount of computation.

[0070] Furthermore, the parameters of the Gaussian difference pyramid include 6 scale levels and 5 octaves.

[0071] In the process of accurately extracting high-quality candidate feature points using the SIFT module, the relevant principles and operational steps for constructing a Gaussian difference pyramid and relocating the scale-space extrema of candidate points are as follows:

[0072] Step 1: Using the candidate point coordinates output by ORB as the center, extract an 11×11 pixel local region, construct a Gaussian pyramid containing O octaves (O value defaults to 5), and each octave contains S scale layers (S value defaults to 6), and perform blurring processing using Gaussian convolution kernels with different standard deviations.

[0073] Step 2: Perform difference operations on adjacent scale layers within the same octave to generate a Difference of Gaussian (DoG) pyramid to highlight potential extreme points;

[0074] Step 3: Delineate a 5×5 pixel detection window centered on the candidate point mapping position, compare the pixels in the window with the surrounding 26 contrast pixels (8 adjacent pixels in the same DoG layer and 18 pixels in the corresponding positions of the upper and lower adjacent DoG layers), and filter out the local gray value maximum or minimum value as potential extreme points.

[0075] Step 4: Perform 3D Taylor expansion fitting on potential extreme points, solve for sub-pixel level offset and iteratively update coordinates until the offset is ≤0.5 pixels. At the same time, remove low-response extreme points with an absolute value of Taylor expansion function <0.03 to achieve sub-pixel level positioning.

[0076] Step 5: Calculate the 2×2 Hessian matrix of the localized extreme point, judge the edge characteristics by the ratio of the matrix trace to the determinant, eliminate edge response points, and finally obtain the accurate scale space extreme point.

[0077] In the process of accurately extracting high-quality candidate feature points, the SURF module uses integral images to calculate the Hessian matrix response around the candidate points and determine the optimal scale. The relevant principles and operation steps are as follows:

[0078] Step 1: For the candidate points output by ORB, a set of continuous scaling parameters σ are preset (corresponding to different degrees of Gaussian blur, such as σ=1.2, 2.4, 3.6…), and a 3×3 Gaussian second-order partial derivative template is constructed for each σ (used to approximate the calculation of Hessian matrix elements).

[0079] Step 2: Quickly calculate the local region pixels of the candidate points at each scale σ using integral images. The calculation of pixels in any rectangular region can be transformed into the addition and subtraction of the pixel values ​​of the four corner points by using integral images. This eliminates the need to traverse pixel by pixel, greatly reducing the amount of computation. Then, based on the convolution of the pixel sum with the corresponding Gaussian template, the three elements of the Hessian matrix are obtained.

[0080] Step 3: According to the Hessian matrix determinant formula Det(H) = Dxx × Dyy − (0.9 × D x y) 2 (0.9 is an empirical coefficient used to balance the weights of each element). Calculate the determinant value of the candidate point at each scale σ. The magnitude of this value reflects the feature response intensity of the candidate point at the corresponding scale (the larger the value, the higher the probability that the point is a feature point).

[0081] Step 4: Compare the Det(H) values ​​of the candidate points at all preset scales, select the scale with the largest Det(H) as the optimal scale of the candidate point, and complete the calculation of the Hessian matrix response based on the integral image and the determination of the optimal scale.

[0082] Furthermore, in step S40, the weights of SIFT and SURF are dynamically adjusted according to the scene texture complexity to generate a fused feature descriptor, specifically including:

[0083] S401. Texture complexity calculation: For each candidate feature point, calculate the pixel gradient variance σ² of its surrounding 3×3 neighborhood. If σ²≥T, it is determined as a high-texture area; if σ²<T, it is determined as a low-texture area.

[0084] S402. Weight assignment: The high-texture area is given a SIFT weight range of 0.6 - 0.7 and a SURF weight range of 0.3 - 0.4; the low-texture area is given a SURF weight range of 0.4 - 0.5 and a SIFT weight range of 0.5 - 0.6.

[0085] S403. Descriptor fusion: Expand the 64-dimensional descriptor of SURF to 128 dimensions by linear interpolation, and perform weighted summation with the 128-dimensional descriptor of SIFT according to the above weights to obtain a 128-dimensional fused descriptor.

[0086] It should be noted that the scene texture complexity is judged based on the pixel gradient variance of the feature point neighborhood, and the weights of SIFT and SURF are dynamically assigned. Those skilled in the art can set the weight value ranges of SIFT and SURF according to actual situations.

[0087] Specifically, the calculation formula for the scene texture complexity includes:

[0088] For each candidate point, calculate the pixel gradient variance σ² of its surrounding 3×3 neighborhood:

[0089] Step 1. Determine the feature point coordinates as ((u, v)), and read the gray values (I(u + i, v + j)) of each pixel in its 3×3 neighborhood (coordinates are ((u + i, v + j)), (i, j ∈ {−1, 0, 1}));

[0090] Step 2. Use a 3×3 Sobel operator to calculate the gradient components of each pixel. The x-direction gradient component (G x =[I(u + 1, v - 1)+2I(u + 1, v)+I(u + 1, v + 1)]−[I(u - 1, v - 1)+2I(u - 1, v)+I(u - 1,

[0091] v + 1)]), where the calculation of the G x direction gradient component is based on the following Sobel_x operator weight matrix:

[0092]

[0093] The calculation formula for the y-direction gradient component is as follows: G y =[I(u - 1, v + 1)+2I(u, v + 1)+I(u + 1, v + 1)]−[I(u - 1, v - 1)+2I(u, v - 1)+I(u + 1, v - 1)]), and it is based on the following Sobel_y operator weight matrix:

[0094] ;

[0095] Step 3. Calculate the gradient magnitude and direction according to the G x , G y values obtained in Step 2. The gradient magnitude can be obtained through ([ ]) (accurate calculation) or (G = |Gx| + |Gy|) (simplified calculation, error ≤ 5%). The gradient direction (θ = arctan2(G ) (accurate calculation) or (G = |Gx| + |Gy|) (simplified calculation, error ≤ 5%). The gradient direction (θ = arctan2(G y , G x) ) (the result range is ([−π, π]), which can be mapped to [0, 2π]). Finally, traverse all 9 pixels in the 3×3 neighborhood to complete the above calculations, and count all gradient magnitudes G. Subsequently, the gradient variance can be calculated based on these G values according to the following formula:

[0096]

[0097] is the average value of the neighborhood gradient magnitudes. Determine the scene texture complexity according to the gradient variance σ 2 to provide a basis for dynamic weighted fusion. If σ² ≥ T (exemplarily, T = 50), it is determined as a high-texture area; if σ² < T, it is determined as a low-texture area.

[0098] Furthermore, extend the SURF descriptor to 128 dimensions through linear interpolation, and then perform weighted summation with the 128-dimensional descriptor of SIFT according to weights to obtain a combined 128-dimensional descriptor. At the same time, the number of ORB extractions will be linked to the scene texture complexity. Set 1 round for high-texture areas and 3 rounds for low-texture areas to ensure the candidate point density.

[0099] Dynamically adjust the weights according to the texture complexity to adapt to various scenarios such as high-texture (industrial parts), low-texture (glass surface), illumination change (indoor / outdoor switching), and target scale change (dynamic vehicle).

[0100] In some embodiments, in step S50, the matching of corresponding points is completed by eliminating mismatches through Euclidean distance matching and RANSAC, including:

[0101] S501. Perform L2 normalization preprocessing on the 128-dimensional combined descriptors generated by the left and right cameras to eliminate the influence of the difference in the numerical ranges of different feature point descriptors on matching;

[0102] S502. For the combined descriptor of each feature point of the left camera, calculate its Euclidean distance from the combined feature descriptors of all feature points of the right camera, and use the nearest neighbor ratio strategy to screen potential corresponding points;

[0103] S503. Remove mismatches using the RANSAC algorithm. The steps for removing mismatches include random sampling, matrix construction, error calculation, and in-point statistics. The steps are iterated 1000 times.

[0104] Select the set of interior points corresponding to the fundamental matrix with the most interior points, determine these interior points as the final exact pairs of corresponding points, complete the matching of corresponding points, and output the exact matching results.

[0105] Furthermore, the nearest neighbor strategy is used to screen potential homonyms, including:

[0106] If the ratio of the nearest neighbor distance between a feature point on the left camera and a feature point on the right camera to the second nearest neighbor distance between the feature point on the left camera and the second nearest neighbor distance between the feature point on the right camera is less than a set distance threshold, then the pair of feature points is identified as potential homonyms, and point pairs with high similarity are initially selected.

[0107] In the above embodiments, both the nearest neighbor distance and the second nearest neighbor distance are Euclidean distances, obtained through the Euclidean distance formula, which is as follows:

[0108]

[0109] Where d LK d RK These are the k-th dimension components of the left and right camera descriptors, respectively.

[0110] For example, the distance setting threshold is set to 0.6.

[0111] In step S503, mismatched points are removed using the RANSAC algorithm. Specifically, this includes: randomly selecting four non-collinear point pairs from potential pairs of identical points; constructing a fundamental matrix (describing the geometric constraints between the left and right camera images) based on the intrinsic and extrinsic parameters of the stereo camera; and calculating the reprojection error of all potential pairs of identical points to this fundamental matrix (using the formula...). ), where (x L ), (x R ) are the homogeneous coordinates of the feature points of the left and right cameras, respectively, and F is the basic matrix. Set the error threshold (default 1.5 pixels) and count the number of point pairs (i.e. the number of inliers) whose reprojection error is less than the threshold.

[0112] Repeat the above process of random sampling, matrix construction, error calculation, and interior point counting, iterating several times (default 1000 iterations), selecting the interior point set corresponding to the basic matrix with the most interior points, and determining these interior points as the final exact pairs of corresponding points, thus completing the pairing of corresponding points.

[0113] In some embodiments, calculating the three-dimensional coordinates of feature points using the triangulation principle includes:

[0114] By using a binocular camera to obtain the two-dimensional projected coordinates of the same feature point from different perspectives, and combining the camera's intrinsic and extrinsic parameters, the three-dimensional spatial coordinates of the feature point are inferred through geometric relationships.

[0115] The intrinsic parameters include focal length f and pixel size dx / dy, while the extrinsic parameters include baseline distance B and relative attitude angle θ. To simplify the calculation, θ is assumed to be 0, meaning the optical axes of the two cameras are parallel.

[0116] It should be further explained that the specific calculation process for calculating the three-dimensional coordinates of feature points using the principle of triangulation is as follows:

[0117] S601. Establish the coordinate system of the stereo camera, and set the optical center of the left camera as the origin O. L The optical center of the right camera is O. R The line connecting the two optical centers is the baseline, with length B. The imaging planes of the left and right cameras are respectively aligned with O. L O R The distance is the focal length f, with O L Establish a three-dimensional coordinate system with the origin as the reference point, and set the X-axis along the baseline direction (from O). L Pointing to O R The Z-axis is perpendicular to the imaging plane (pointing towards the scene), and the Y-axis is perpendicular to the XZ plane;

[0118] S602. Obtain the two-dimensional pixel coordinates of the corresponding feature points on the left and right camera images. Let the pixel coordinates of the left camera be (u L ,v L The right camera is (u) R ,v R );

[0119] S603. Convert pixel coordinates to imaging plane coordinates in the camera coordinate system, left camera imaging plane coordinates (x... L ,y L )=((u L -u0)×dx,(v L (−v0)×dy), right camera imaging plane coordinates (x) R ,y R )=((u R -u0)×dx,(v R (−v0)×dy) (where (u0,v0) are the pixel coordinates of the camera principal point);

[0120] S604. Derive the 3D coordinates based on the principle of similar triangles. Since the optical axes of the two cameras are parallel, the feature point P(X, Y, Z) lies in the coordinate system of the left camera, satisfying x L / f=X / Z、y L / f=Y / Z; In the right camera coordinate system (X'=XB, Y'=Y, Z'=Z), x satisfies R / f=(X−B) / Z;

[0121] Solving the simultaneous equations yields the three-dimensional coordinates of the feature point: X = (B × x) L ) / (x L -x R ), Y=(B×y L ) / (x L -x R Z = (B × f) / (x) L -x R (where x) L -x R Parallax is the difference in X-direction coordinates of corresponding points on the imaging planes of two cameras.

[0122] The three-dimensional spatial coordinates of the feature points are obtained through the above calculation equations for X, Y, and Z, thus completing the conversion from two-dimensional coordinates to three-dimensional coordinates under binocular vision.

[0123] In summary, the 3D coordinate calculation process combines the intrinsic and extrinsic parameters of the binocular camera with the 2D coordinates of matched corresponding points, and uses the triangulation principle to calculate the 3D coordinates (X, Y, Z) of feature points, thus completing the 2D-to-3D coordinate transformation. This transformation method addresses the speed issue through ORB pre-screening, the accuracy issue through SIFT / SURF precise extraction, and the scene adaptation issue through dynamic weighting, thereby improving transformation accuracy and real-time performance, significantly enhancing scene adaptability, and reducing engineering implementation costs.

[0124] It should be noted that the abbreviations and key terms involved in this invention are defined as follows:

[0125] SIFT (Scale-Invariant Feature Transform) is a feature point extraction algorithm with scale and rotation invariance. It detects extreme points through the difference of Gaussian pyramid and generates 128-dimensional feature descriptors.

[0126] SURF (Speeded Up Robust Features): An improved algorithm of SIFT, it accelerates computation based on integral images and generates 64-dimensional feature descriptors, balancing speed and robustness.

[0127] ORB (Oriented FAST and Rotated BRIEF): A rotation-invariant algorithm for fast feature point extraction. It combines FAST corner detection with rotation-invariant BRIEF descriptors to generate 256-bit binary descriptors, resulting in fast extraction speed.

[0128] Binocular vision: A technique that simulates the principle of human binocular vision by using two cameras to acquire images from different perspectives and using the principle of triangulation to calculate the three-dimensional coordinates of points in space.

[0129] Feature point matching is the process of finding "same-name feature points" (projections of the same physical point) in images from different viewpoints. It is the core step in 2D-to-3D coordinate transformation.

[0130] RANSAC (Random Sample Consensus): A random sampling consensus algorithm used to eliminate mismatches in feature point matching and improve the reliability of matching results.

[0131] Triangulation is a mathematical method for calculating the three-dimensional spatial coordinates of a point using the intrinsic and extrinsic parameters of a binocular camera and the two-dimensional coordinates of corresponding feature points.

[0132] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-described technical content to create equivalent embodiments without departing from the scope of the present invention. The implementation schemes in the above embodiments can also be further combined or replaced. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A binocular vision 2D-3D coordinate transformation method based on multi-feature fusion, characterized in that, It includes: S10. Initialization settings, setting binocular camera parameters, image resolution, ORB extraction parameters, SIFT / SURF extraction parameters, and fusion weight calculation threshold; S20. Input images to the left and right cameras in the binocular camera, and quickly extract candidate feature points using the ORB algorithm to obtain the two-dimensional coordinates and preliminary rotation directions of high-quality candidate feature points; S30. Synchronously start the SIFT module and the SURF module to accurately extract the high-quality candidate feature points obtained in step S20; S40. Dynamically adjust the weights of SIFT and SURF according to the scene texture complexity to generate a fused feature descriptor; S50. For the fused feature point sets of the left and right cameras obtained in step S40, complete the pairing of corresponding points by removing mismatches through Euclidean distance matching and RANSAC, and output the accurate matching result; S60. Combine the internal and external parameters of the binocular camera and the two-dimensional coordinates of the matched corresponding points, and use the principle of triangulation to calculate the three-dimensional coordinates of the feature points to complete the two-dimensional to three-dimensional coordinate transformation.

2. The binocular vision 2D-3D coordinate transformation method based on multi-feature fusion according to claim 1, characterized in that, The binocular camera parameters include internal and external parameters, where the internal parameters include the focal length f, pixel size dx / dy, and the external parameters include the baseline distance B and relative attitude angle θ; the fusion weight calculation threshold includes the texture complexity threshold T.

3. The binocular vision 2D-3D coordinate transformation method based on multi-feature fusion according to claim 1, characterized in that, In step S20, the ORB algorithm is used to quickly extract candidate feature points, including: Optimizing the candidate set through contrast screening and edge elimination, excluding invalid points with low contrast and edge interference, and outputting the two-dimensional coordinates (u, v) and preliminary rotation direction of high-quality candidate feature points.

4. The binocular vision two-dimensional to three-dimensional coordinate transformation method based on multi-feature fusion according to any one of claims 1-3, characterized in that, [[ID=!0]]In step 30, [[ID=!1]]The accurate extraction of high-quality candidate feature points by the SIFT module includes: constructing a difference-of-Gaussians pyramid, repositioning the scale-space extreme points of the candidate points, generating a 128-dimensional high-precision feature descriptor, and recording the scale information and response intensity; [[ID=!2]]The accurate extraction of high-quality candidate feature points by the SURF module includes: calculating the response of the Hessian matrix around the candidate points using the integral image, determining the optimal scale, generating a 64-dimensional robust feature descriptor, and recording the contrast and response intensity.

5. The binocular vision 2D-3D coordinate transformation method based on multi-feature fusion according to claim 4, characterized in that, [[ID=!3]]The parameters of the difference-of-Gaussians pyramid include 6 scale layers and 5 octaves.

6. The binocular vision 2D-3D coordinate transformation method based on multi-feature fusion according to claim 4, characterized in that, [[ID=!4]]Step S40 dynamically adjusts the weights of SIFT and SURF according to the scene texture complexity to generate a fused feature descriptor, specifically including: [[ID=!5]]S401. Texture complexity calculation: For each candidate feature point, calculate the pixel gradient variance σ² of its 3×3 neighborhood. If σ²≥T, it is determined as a high-texture area; if σ²<T, it is determined as a low-texture area; [[ID=!6]]S402. Weight assignment: The high-texture area is given a SIFT weight range of 0.6 - 0.7 and a SURF weight range of 0.3 - 0.4; the low-texture area is given a SURF weight range of 0.4 - 0.5 and a SIFT weight range of 0.5 - 0.6; [[ID=!7]]S403. Descriptor fusion: Expand the 64-dimensional descriptor of SURF to 128 dimensions through linear interpolation, and perform weighted summation with the 128-dimensional descriptor of SIFT according to the above weights to obtain a 128-dimensional fused descriptor.

7. The binocular vision two-dimensional to three-dimensional coordinate transformation method based on multi-feature fusion according to any one of claims 1-3, characterized in that, Step S50 completes the matching of corresponding points through Euclidean distance matching and RANSAC mismatch removal, including: S501. Perform L2 normalization preprocessing on the 128-dimensional fusion descriptors generated by the left and right cameras to eliminate the influence of differences in the numerical range of different feature point descriptors on matching. S502. For each feature point of the left camera, calculate its Euclidean distance with the fused feature descriptors of all feature points of the right camera, and use the nearest neighbor comparison strategy to screen potential homonyms. S503. Remove mismatches using the RANSAC algorithm. The steps for removing mismatches include random sampling, matrix construction, error calculation, and in-point statistics. These steps are iterated 1000 times. Select the set of interior points corresponding to the fundamental matrix with the most interior points, determine these interior points as the final exact pairs of corresponding points, complete the matching of corresponding points, and output the exact matching results.

8. The binocular vision 2D-3D coordinate transformation method based on multi-feature fusion according to claim 7, characterized in that, The method of using the nearest neighbor comparison strategy to screen potential homonyms includes: If the ratio of the nearest neighbor distance between a feature point on the left camera and a feature point on the right camera to the second nearest neighbor distance between the feature point on the left camera and the second nearest neighbor distance between the feature point on the right camera is less than a set distance threshold, then the pair of feature points is identified as potential homonyms, and point pairs with high similarity are initially selected.

9. The binocular vision 2D-3D coordinate transformation method based on multi-feature fusion according to claim 8, characterized in that, The nearest neighbor distance and the second nearest neighbor distance are both Euclidean distances, obtained using the Euclidean distance calculation formula, which is as follows: ； Where d LK d RK These are the k-th dimension components of the left and right camera descriptors, respectively.

10. The binocular vision two-dimensional to three-dimensional coordinate transformation method based on multi-feature fusion according to any one of claims 1-3, characterized in that, Calculating the three-dimensional coordinates of feature points using the aforementioned triangulation principle includes: By using a binocular camera to obtain the two-dimensional projected coordinates of the same feature point from different perspectives, and combining the camera's intrinsic and extrinsic parameters, the three-dimensional spatial coordinates of the feature point are inferred through geometric relationships.

Images