A polar line correction method and device

Abstract

The present invention provides an epipolar correction method and device, the method comprising: acquiring a first image and a second image based on a binocular vision system; determining a plurality of first feature points in the first image and a plurality of second feature points in the second image based on the grayscale values ​​of image pixels; matching the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs; determining a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and a plurality of initial matching point pairs; determining a first pole and a second pole based on the target basic matrix and an epipolar constraint, and determining a first projection matrix and a second projection matrix based on a center point, a first pole, and a second pole of the first image; performing a projection transformation on the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain a corrected image. The present invention improves the accuracy and efficiency of epipolar correction.

Description

Technical Field

[0001] The present invention relates to the technical field of image matching, and in particular to an epipolar line correction method and device. Background Art

[0002] Binocular stereo matching is currently a hot research topic with a wide range of practical applications. Most of the matching algorithms are based on the assumption that the feature points corresponding to the two images obtained by the binocular stereo system are on the same scan line. This reduces the matching search range from two dimensions to one dimension, speeding up the matching speed and accuracy. However, in reality, due to the influence of the camera structure and manufacturing process, it is difficult to ensure that the main axes of the camera are parallel, so the binocular system is no longer parallel. Therefore, a method of correcting the binocular system to make it parallel appears, which is epipolar correction.

[0003] Most of the epipolar correction methods in the prior art are based on the projection transformation matrix. This method is a method that completely relies on the basic matrix and has the following technical problems: it is overly dependent on the accuracy of the basic matrix. When there is a mismatch, the epipolar correction result will be inaccurate, causing large image distortion and reducing the matching accuracy.

[0004] Therefore, it is urgent to propose a method and device for epipolar correction to improve the accuracy of the basic matrix, thereby improving the accuracy of epipolar correction. Summary of the invention

[0005] In view of this, it is necessary to provide an epipolar line correction method and device to solve the technical problem in the prior art that when the matching accuracy of the basic matrix is ​​not high, the epipolar line correction accuracy is low.

[0006] In one aspect, the present invention provides a polar line correction method, comprising:

[0007] Acquire a first image and a second image based on a binocular vision system;

[0008] Determine a plurality of first feature points in the first image and a plurality of second feature points in the second image based on grayscale values ​​of image pixels;

[0009] Matching the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs;

[0010] Determining a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and the plurality of initial matching point pairs;

[0011] Determine a first pole and a second pole based on the target fundamental matrix and the epipolar constraint, and determine a first projection matrix and a second projection matrix based on a center point of the first image, the first pole, and the second pole;

[0012] Projection transformation is performed on the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain a corrected image.

[0013] In some possible implementations, determining a plurality of first feature points in the first image and a plurality of second feature points in the second image based on grayscale values ​​of image pixels includes:

[0014] Performing smoothing processing on the first image and the second image to obtain a first smoothed image and a second smoothed image respectively;

[0015] Performing multi-scale decomposition on the first smoothed image and the second smoothed image to obtain a first Gaussian pyramid and a second Gaussian pyramid respectively;

[0016] Dividing the first Gaussian pyramid and the second Gaussian pyramid based on a preset division rule, and correspondingly obtaining a plurality of first sub-image blocks and a plurality of second sub-image blocks;

[0017] A local sliding window is constructed according to the first sub-image block or the second sub-image block, and the multiple first feature points are determined according to the local sliding window and the multiple first sub-image blocks, and the multiple second feature points are determined according to the local sliding window and the multiple second sub-image blocks.

[0018] In some possible implementations, the first Gaussian pyramid includes a plurality of first image layers, and dividing the first Gaussian pyramid based on a preset division rule to obtain a plurality of first sub-image blocks includes:

[0019] Dividing each of the plurality of first image layers to obtain a plurality of first image regions;

[0020] Each of the multiple first image areas is divided to obtain multiple first sub-image blocks.

[0021] In some possible implementations, the multiple first image layers include a current first image layer and a first adjacent image layer and a second adjacent image layer adjacent to the current first image layer; the multiple first sub-image blocks include a current first sub-image block and a multiple adjacent first sub-image blocks adjacent to the current first sub-image block; and determining the multiple first feature points according to the local sliding window and the multiple first sub-image blocks includes:

[0022] Acquire the original image grayscale of the current first sub-image block in the local sliding window;

[0023] Moving the local sliding window to traverse the current first sub-image block, and acquiring the current image grayscale of the current first sub-image block in the moved local sliding window;

[0024] Determine a grayscale difference sum based on the original image grayscale and the current image grayscale;

[0025] When the grayscale difference is greater than or equal to the difference threshold, taking the pixel with the largest grayscale value in the current first sub-image block as a key point;

[0026] Determine a first adjacent point in the first adjacent image layer corresponding to the key point and a second adjacent point in the second adjacent image layer corresponding to the key point;

[0027] When the grayscale value of the key point is greater than the grayscale value of the first adjacent point, the grayscale value of the key point is greater than the grayscale value of the second adjacent point, and the grayscale value of the key point is greater than the grayscale value of each pixel in the multiple adjacent first sub-image blocks, the key point is the first feature point.

[0028] In some possible implementations, matching the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs includes:

[0029] Determine the Euclidean distance between each first feature point in the plurality of first feature points and each second feature point in the plurality of second feature points;

[0030] It is determined whether the Euclidean distance is greater than a preset distance. When the Euclidean distance is greater than the preset distance, the first feature point and the second feature point are the initial matching point pair.

[0031] In some possible implementations, determining a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and the plurality of initial matching point pairs includes:

[0032] Step 1: randomly determine eight first initial matching point pairs from the multiple initial matching point pairs, and determine an initial basic matrix and epipolar lines according to the eight first initial matching point pairs;

[0033] Step 2: determining the distances between a plurality of second initial matching point pairs other than the first initial matching point pair among the plurality of initial matching point pairs and the epipolar line;

[0034] Step 3: determine whether the distance is greater than a threshold distance. When the distance is less than the threshold distance, the second initial matching point pair is an inner point. When the distance is greater than or equal to the threshold distance, the second initial matching point pair is an outer point.

[0035] Step 4: Determine whether the ratio of the number of inliers to the number of the multiple initial matching point pairs is greater than a threshold ratio and whether the number of iterations is greater than a maximum number of iterations. If the ratio of the number of inliers to the number of the multiple initial matching point pairs is greater than the threshold ratio, or the number of iterations is greater than the maximum number of iterations, the initial basic matrix is ​​the target basic matrix. If the ratio of the number of inliers to the number of the multiple initial matching point pairs is less than or equal to the threshold ratio, and the number of iterations is less than or equal to the maximum number of iterations, repeat steps 1 to 4.

[0036] In some possible implementations, determining the first projection matrix and the second projection matrix based on the center point of the first image, the first pole, and the second pole includes:

[0037] Performing homogeneous transformation and simplification on the center point of the first image to obtain an initial translation transformation matrix;

[0038] Performing rotation mapping on the first image and the second image to obtain a first rotation matrix and a second rotation matrix respectively;

[0039] Project the first pole and the second pole to a point at infinity in the horizontal direction, and correspondingly obtain a first standard projection transformation matrix and a second standard projection transformation matrix;

[0040] Performing translation transformation on the first image and the second image to obtain a target translation transformation matrix;

[0041] Determine the first projection matrix based on the initial translation transformation matrix, the first rotation matrix, the first standard projection transformation matrix, and the target translation transformation matrix;

[0042] The second projection matrix is ​​determined based on the initial translation transformation matrix, the second rotation matrix, and the second standard projection transformation matrix.

[0043] In some possible implementations, the epipolar line correction method further includes:

[0044] Optimize the first projection matrix and the second projection matrix based on the optimization model to obtain a first optimized projection matrix and a second optimized projection matrix;

[0045] The projecting transformation of the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain the corrected image is specifically:

[0046] The first feature points and the second feature points are projected and transformed based on the first optimized projection matrix and the second optimized projection matrix to obtain a corrected image.

[0047] In some possible implementations, the initial matching point pair includes a first matching point and a second matching point, and the optimization model is:

[0048]

[0049] In the formula, S is the optimization target; H 0 is the first projection matrix; H 1 is the second projection matrix; u 0i is the coordinate value of the first matching point; u 1i is the coordinate value of the second matching point; N is the number of initial matching point pairs.

[0050] On the other hand, the present invention also provides a polar line correction device, comprising:

[0051] An image acquisition unit, configured to acquire a first image and a second image based on a binocular vision system;

[0052] a feature point determination unit, configured to determine a plurality of first feature points in the first image and a plurality of second feature points in the second image based on grayscale values ​​of image pixels;

[0053] A matching point determination unit, used for matching the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs;

[0054] A basic matrix determination unit, used to determine a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and the plurality of initial matching point pairs;

[0055] a projection matrix determining unit, configured to determine a first pole and a second pole based on the target base matrix and the epipolar constraint, and to determine a first projection matrix and a second projection matrix based on a center point of the first image, the first pole and the second pole;

[0056] An image correction unit is used to perform projection transformation on the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain a corrected image.

[0057] The beneficial effect of adopting the above embodiment is that the epipolar correction method provided by the present invention can eliminate the mismatched points in the multiple initial matching point pairs by setting the target basic matrix based on the normalized eight-point algorithm, the random consistency algorithm and multiple initial matching point pairs, and improve the accuracy of the obtained target basic matrix, thereby improving the accuracy of the epipolar correction. In addition, since the random consistency algorithm is a random process and has robustness, it can estimate a high-precision target basic matrix from multiple initial matching point pairs, which can further improve the accuracy of the epipolar correction.

[0058] Furthermore, the present invention can improve the acquisition speed of the first feature points and the second feature points, thereby improving the efficiency of epipolar line correction by setting a plurality of first feature points in the first image and a plurality of second feature points in the second image based on the grayscale values ​​of image pixels.

[0059] Furthermore, the present invention determines the target basic matrix based on the normalized eight-point algorithm. Since the normalized eight-point algorithm is a linear solution with fast calculation speed, it can improve the efficiency of obtaining the target basic matrix, thereby improving the efficiency of epipolar correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 A schematic flow chart of an embodiment of the epipolar line correction method provided by the present invention;

[0062] Figure 2 For the present invention Figure 1 A schematic flow chart of an embodiment of S102;

[0063] Figure 3 For the present invention Figure 2 A schematic flow chart of an embodiment of S203;

[0064] Figure 4 For the present invention Figure 2 A schematic flow chart of an embodiment of S204;

[0065] Figure 5 For the present invention Figure 1 A schematic flow chart of an embodiment of S103;

[0066] Figure 6 For the present invention Figure 1 A schematic flow chart of an embodiment of S104;

[0067] Figure 7 For the present invention Figure 1 A schematic flow chart of an embodiment of S105;

[0068] Figure 8 A schematic diagram of the structure of an embodiment of the polar line correction device provided by the present invention;

[0069] Fig. 9 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0071] It should be understood that the schematic drawings are not drawn to scale. The flow chart used in the present invention shows the operations implemented according to some embodiments of the present invention. It should be understood that the operations of the flow chart can be implemented out of order, and the steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flow chart under the guidance of the content of the present invention, and can also remove one or more operations from the flow chart.

[0072] Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0073] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0074] The embodiments of the present invention provide a polar line correction method and device, which are described below respectively.

[0075] Figure 1 A schematic flow chart of an embodiment of the epipolar line correction method provided by the present invention is shown in FIG. Figure 1 As shown, the epipolar correction method includes:

[0076] S101, acquiring a first image and a second image based on a binocular vision system;

[0077] S102, determining a plurality of first feature points in the first image and a plurality of second feature points in the second image based on the grayscale values ​​of the image pixels;

[0078] S103, matching the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs;

[0079] S104, determining a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and a plurality of initial matching point pairs;

[0080] S105, determining a first pole and a second pole based on the target basic matrix and the epipolar constraint, and determining a first projection matrix and a second projection matrix based on the center point, the first pole and the second pole of the first image;

[0081] S106 , performing projection transformation on the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain a corrected image.

[0082] Compared with the prior art, the epipolar correction method provided by the embodiment of the present invention can eliminate mismatched points in multiple initial matching point pairs by setting a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and multiple initial matching point pairs, thereby improving the accuracy of the obtained target basic matrix and thus improving the accuracy of epipolar correction. In addition, since the random consistency algorithm is a random process and has robustness, it can estimate a high-precision target basic matrix from multiple initial matching point pairs, which can further improve the accuracy of epipolar correction.

[0083] Furthermore, the embodiments of the present invention can improve the acquisition speed of the first feature points and the second feature points by setting a plurality of first feature points in the first image and a plurality of second feature points in the second image based on the grayscale values ​​of the image pixels, thereby improving the efficiency of the epipolar line correction.

[0084] Furthermore, the embodiment of the present invention determines the target basic matrix based on the normalized eight-point algorithm. Since the normalized eight-point algorithm is a linear solution with a fast operation speed, the efficiency of obtaining the target basic matrix can be improved, thereby improving the efficiency of epipolar correction.

[0085] In some embodiments of the present invention, Figure 2 As shown, step S102 includes:

[0086] S201, performing smoothing processing on the first image and the second image to obtain a first smoothed image and a second smoothed image respectively;

[0087] S202, performing multi-scale decomposition on the first smoothed image and the second smoothed image to obtain a first Gaussian pyramid and a second Gaussian pyramid accordingly;

[0088] S203, dividing the first Gaussian pyramid and the second Gaussian pyramid based on a preset division rule, and correspondingly obtaining a plurality of first sub-image blocks and a plurality of second sub-image blocks;

[0089] S204: construct a local sliding window according to the first sub-image block or the second sub-image block, and determine a plurality of first feature points according to the local sliding window and the plurality of first sub-image blocks, and determine a plurality of second feature points according to the local sliding window and the plurality of second sub-image blocks.

[0090] The embodiment of the present invention can make the obtained first smoothed image and the second smoothed image more stable by smoothing the first image and the second smoothed image, thereby improving the robustness of the obtained first smoothed image and the second smoothed image, thereby improving the robustness of the epipolar correction method.

[0091] Among them, step S201 is specifically: first use the morphological opening operation method to corrode and then expand the first image and the second image to preliminarily filter out the noise existing in the first image and the second image, and then use Gaussian filtering to further filter out the noise points in the first image and the second image to obtain a relatively stable first smooth image and a second smooth image.

[0092] Specifically, the Gaussian filter function is:

[0093]

[0094] Wherein, G(x,y) is the pixel value of each pixel in the first smoothed image or the second smoothed image; x is the horizontal coordinate of the pixel in the first image or the second image; y is the vertical coordinate of the pixel in the first image or the second image; σ is the pixel standard deviation of each pixel in the first image or the second image.

[0095] It should be noted that: the first Gaussian pyramid includes multiple first image layers, and the second Gaussian pyramid includes multiple second image layers, wherein the multiple first image layers and the multiple second image layers are different scale spaces caused by imitating the distance of human vision, and the scales between the multiple first image layers and the multiple second image layers are consistent. Since the number of pixels in the image layer decreases as the scale increases, the first Gaussian pyramid includes 6 first image layers, and the second Gaussian pyramid includes 6 second image layers.

[0096] It should also be noted that the local sliding window in step S204 includes but is not limited to a rectangular window, a Gaussian window, etc.

[0097] In some embodiments of the present invention, Figure 3 As shown, step S203 includes:

[0098] S301, dividing each of a plurality of first image layers to obtain a plurality of first image regions;

[0099] S302: Divide each of the multiple first image areas to obtain multiple first sub-image blocks.

[0100] In a specific embodiment of the present invention, step S301 is specifically: divide each first image layer into 3*3 nine-square grid areas to obtain 9 first image areas. Step S302 is specifically: divide each first image area into 4*4 small areas to obtain multiple first image blocks.

[0101] It should be understood that the method for acquiring the second sub-image block is the same as the method for acquiring the first sub-image block, see steps S301-S302 for details, and will not be described in detail here.

[0102] In some embodiments of the present invention, the plurality of first image layers include a current first image layer and a first adjacent image layer and a second adjacent image layer adjacent to the current first image layer, and the plurality of first sub-image blocks include a current first sub-image block and a plurality of adjacent first sub-image blocks adjacent to the current first sub-image block. Figure 4 As shown, step S204 includes:

[0103] S401, obtaining the original image grayscale of the current first sub-image block in the local sliding window;

[0104] S402, moving the local sliding window to traverse the current first sub-image block, and obtaining the current image grayscale of the current first sub-image block in the moved local sliding window;

[0105] S403, determining a grayscale difference sum based on the grayscale of the original image and the grayscale of the current image;

[0106] S404, when the grayscale difference is greater than or equal to the difference threshold, taking the pixel with the largest grayscale value in the current first sub-image block as a key point;

[0107] S405, determining a first adjacent point corresponding to the key point in the first adjacent image layer and a second adjacent point corresponding to the key point in the second adjacent image layer;

[0108] S406: When the grayscale value of the key point is greater than the grayscale value of the first adjacent point, the grayscale value of the key point is greater than the grayscale value of the second adjacent point, and the grayscale value of the key point is greater than the grayscale value of each pixel in multiple adjacent first sub-image blocks, the key point is a first feature point.

[0109] Specifically, the grayscale difference sum in step S403 is:

[0110] E(u,v)=∑ x,y w(x,y)[I(x+u,y+v)-I(x,y)] 2

[0111] Where E(u,v) is the grayscale difference; w(x,y) is the window function; (u,v) is the movement of the local sliding window; I(x+u,y+v) is the current image grayscale; I(x,y) is the original image grayscale.

[0112] It should be noted that the difference threshold in step S404 can be adjusted or set according to actual application scenarios and is not specifically limited here.

[0113] It should be understood that the method for acquiring the second feature point is the same as the method for acquiring the first feature point, see steps S401 to S406 for details, and will not be described in detail here.

[0114] In some embodiments of the present invention, Figure 5 As shown, step S103 includes:

[0115] S501, determining the Euclidean distance between each first feature point in a plurality of first feature points and each second feature point in a plurality of second feature points;

[0116] S502: Determine whether the Euclidean distance is greater than a preset distance. When the Euclidean distance is greater than the preset distance, the first feature point and the second feature point are an initial matching point pair.

[0117] It should be noted that the preset distance can be set or adjusted according to the actual application scenario and is not specifically limited here.

[0118] In some embodiments of the present invention, Figure 6 As shown, step S104 includes:

[0119] S601, randomly determining eight first initial matching point pairs from a plurality of initial matching point pairs, and determining an initial basic matrix and epipolar lines according to the eight first initial matching point pairs;

[0120] S602, determining distances between a plurality of second initial matching point pairs other than the first initial matching point pair among the plurality of initial matching point pairs and the epipolar line;

[0121] S603, determining whether the distance is greater than a threshold distance, when the distance is less than the threshold distance, the second initial matching point pair is an inner point, when the distance is greater than or equal to the threshold distance, the second initial matching point pair is an outer point;

[0122] S604, determine whether the ratio of the number of inner points to the number of multiple initial matching point pairs is greater than the threshold ratio and whether the number of iterations is greater than the maximum number of iterations. If the ratio of the number of inner points to the number of multiple initial matching point pairs is greater than the threshold ratio, or the number of iterations is greater than the maximum number of iterations, the initial basic matrix is ​​the target basic matrix. If the ratio of the number of inner points to the number of multiple initial matching point pairs is less than or equal to the threshold ratio, and the number of iterations is less than or equal to the maximum number of iterations, repeat steps S601 to S604.

[0123] It should be understood that the threshold distance, the threshold ratio and the maximum number of iterations can be set or adjusted according to actual application scenarios. In a specific embodiment of the present invention, the threshold ratio is 95%.

[0124] In some embodiments of the present invention, determining the first pole and the second pole based on the target basic matrix and the epipolar constraint in step S105 is specifically:

[0125] From the epipolar constraints we know that:

[0126] Fe 0 =0 F T e 1 =0

[0127] Where F is the target basic matrix; e 0 is the first pole, e 1 The second pole.

[0128] Then, the first pole and the second pole can be obtained according to the above-mentioned polar line constraint and the target basic matrix.

[0129] In some embodiments of the present invention, Figure 7 As shown, step S105 includes:

[0130] S701, performing homogeneous transformation and simplification on the center point of the first image to obtain an initial translation transformation matrix;

[0131] S702, performing rotation mapping on the first image and the second image to obtain a first rotation matrix and a second rotation matrix accordingly;

[0132] S703, projecting the first pole and the second pole to a point at infinity in the horizontal direction, and correspondingly obtaining a first standard projection transformation matrix and a second standard projection transformation matrix;

[0133] S704, performing translation transformation on the first image and the second image to obtain a target translation transformation matrix;

[0134] S705, determining a first projection matrix based on the initial translation transformation matrix, the first rotation matrix, the first standard projection transformation matrix, and the target translation transformation matrix;

[0135] S706: Determine a second projection matrix based on the initial translation transformation matrix, the second rotation matrix, and the second standard projection transformation matrix.

[0136] Step S701 specifically includes: transforming the coordinates of the center point of the first image to (a 0 , b 0 , 1), to avoid unnecessary trouble, (a 0 , b 0 ,1) is simplified to (0, 0, 1), then the initial translation transformation matrix T is:

[0137]

[0138] Step S702 is specifically as follows: project the first pole and the second pole to infinity, first perform rotation mapping on the first image and the second image, and the rotation mapping is α and 0 and α 1 , let the first pole and the second pole fall on the x-axis, then the first rotation matrix R 0 and the second rotation matrix R 1 They are:

[0139]

[0140]

[0141] The first standard projection transformation matrix M in step S703 0 and the second standard projection transformation matrix M 1 They are:

[0142]

[0143]

[0144] In the formula, f 1 is the image perspective deformation of the first image; f 2 is the image perspective deformation of the second image.

[0145] After step S703, the epipolar lines are parallel, but not on the same scanning line, and there is a vertical time difference. Therefore, the first image and the second image need to be translated. Assuming the moving component of the ordinate is l, the target translation transformation matrix in step S704 is:

[0146]

[0147] Then the first projection matrix H in step S705 is 0 for:

[0148] H 0=NM 0 R 0 T

[0149] The second projection matrix H in step S706 1 for:

[0150] H 1 =M 1 R 1 T

[0151] The first feature point and the second feature point can be corrected by using the first projection matrix and the second projection matrix to obtain a corrected image.

[0152] In order to further improve the correction accuracy, in some embodiments of the present invention, after step S105, the epipolar line correction method further includes:

[0153] Optimize the first projection matrix and the second projection matrix based on the optimization model to obtain a first optimized projection matrix and a second optimized projection matrix;

[0154] Then step S106 specifically includes: performing projection transformation on the first feature point and the second feature point based on the first optimized projection matrix and the second optimized projection matrix to obtain a corrected image.

[0155] The embodiment of the present invention can further improve the accuracy of the first projection matrix and the second projection matrix by optimizing the first projection matrix and the second projection matrix, thereby further improving the accuracy of the obtained corrected image.

[0156] In a specific embodiment of the present invention, the initial matching point pair includes a first matching point and a second matching point, and the optimization model is:

[0157]

[0158] In the formula, S is the optimization target; H 0 is the first projection matrix; H 1 is the second projection matrix; u 0i is the coordinate value of the first matching point; u 1i is the coordinate value of the second matching point; N is the number of initial matching point pairs.

[0159] The specific optimization process is: minimize the optimization target, that is, S=0, and solve the above optimization model using the constraint condition that the ordinates of the corrected matching points are the same to obtain the first optimized projection matrix and the second optimized projection matrix.

[0160] In order to better implement the epipolar line correction method in the embodiment of the present invention, based on the epipolar line correction method, the embodiment of the present invention also provides an epipolar line correction device, such as Figure 8 As shown, the polar line correction device 800 includes:

[0161] An image acquisition unit 801 is used to acquire a first image and a second image based on a binocular vision system;

[0162] A feature point determination unit 802, configured to determine a plurality of first feature points in the first image and a plurality of second feature points in the second image based on grayscale values ​​of image pixels;

[0163] A matching point determination unit 803, configured to match the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs;

[0164] A basic matrix determining unit 804 is used to determine a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and a plurality of initial matching point pairs;

[0165] A projection matrix determining unit 805 is used to determine the first pole and the second pole based on the target basic matrix and the epipolar constraint, and to determine the first projection matrix and the second projection matrix based on the center point, the first pole and the second pole of the first image;

[0166] The image correction unit 806 is used to perform projection transformation on the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain a corrected image.

[0167] The epipolar line correction device 800 provided in the above embodiment can implement the technical solution described in the above epipolar line correction method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above epipolar line correction method embodiment, which will not be repeated here.

[0168] like Fig. 9 As shown, the present invention also provides an electronic device 900. The electronic device 900 includes a processor 901, a memory 902 and a display 903. Fig. 9 Only some components of the electronic device 900 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0169] In some embodiments, the processor 901 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 902, such as the polar line correction method of the present invention.

[0170] In some embodiments, the processor 901 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processor 901 may be local or remote. In some embodiments, the processor 901 may be implemented in a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0171] In some embodiments, the memory 902 may be an internal storage unit of the electronic device 900, such as a hard disk or memory of the electronic device 900. In other embodiments, the memory 902 may also be an external storage device of the electronic device 900, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 900.

[0172] Furthermore, the memory 902 may include both an internal storage unit of the electronic device 900 and an external storage device. The memory 902 is used to store application software installed in the electronic device 900 and various data.

[0173] In some embodiments, the display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 903 is used to display information on the electronic device 900 and to display a visual user interface. The components 901-903 of the electronic device 900 communicate with each other via a system bus.

[0174] In some embodiments of the present invention, when the processor 901 executes the polar line correction program in the memory 902, the following steps may be implemented:

[0175] Acquire a first image and a second image based on a binocular vision system;

[0176] Determine a plurality of first feature points in the first image and a plurality of second feature points in the second image based on the grayscale values ​​of the image pixels;

[0177] Matching the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs;

[0178] Determine the target basic matrix based on the normalized eight-point algorithm, the random consistency algorithm and multiple initial matching point pairs;

[0179] Determine a first pole and a second pole based on the target fundamental matrix and the epipolar constraint, and determine a first projection matrix and a second projection matrix based on the center point of the first image, the first pole and the second pole;

[0180] Projection transformation is performed on the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain a corrected image.

[0181] It should be understood that: when the processor 901 executes the polar line correction program in the memory 902, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0182] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 900 mentioned, and the electronic device 900 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 900 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0183] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the steps or functions in the polar line correction method provided in the above-mentioned method embodiments can be implemented.

[0184] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0185] The polar line correction method and device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A polar line correction method, characterized in that: include: Acquire a first image and a second image based on a binocular vision system; Determine a plurality of first feature points in the first image and a plurality of second feature points in the second image based on grayscale values ​​of image pixels; Matching the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs; Determining a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and the plurality of initial matching point pairs; Determine a first pole and a second pole based on the target fundamental matrix and the epipolar constraint, and determine a first projection matrix and a second projection matrix based on a center point of the first image, the first pole, and the second pole; Performing projection transformation on the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain a corrected image; The determining of a plurality of first feature points in the first image and a plurality of second feature points in the second image based on the grayscale values ​​of the image pixels comprises: Performing smoothing processing on the first image and the second image to obtain a first smoothed image and a second smoothed image respectively; Performing multi-scale decomposition on the first smoothed image and the second smoothed image to obtain a first Gaussian pyramid and a second Gaussian pyramid respectively; Dividing the first Gaussian pyramid and the second Gaussian pyramid based on a preset division rule, and correspondingly obtaining a plurality of first sub-image blocks and a plurality of second sub-image blocks; constructing a local sliding window according to the first sub-image block or the second sub-image block, and determining the plurality of first feature points according to the local sliding window and the plurality of first sub-image blocks, and determining the plurality of second feature points according to the local sliding window and the plurality of second sub-image blocks; The step of matching the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs includes: Determine the Euclidean distance between each first feature point in the plurality of first feature points and each second feature point in the plurality of second feature points; Determine whether the Euclidean distance is greater than a preset distance, and when the Euclidean distance is greater than the preset distance, the first feature point and the second feature point are the initial matching point pair; The determining of the target basic matrix based on the normalized eight-point algorithm, the random consistency algorithm and the multiple initial matching point pairs includes: Step 1: randomly determine eight first initial matching point pairs from the multiple initial matching point pairs, and determine an initial basic matrix and epipolar lines according to the eight first initial matching point pairs; Step 2: determining the distances between a plurality of second initial matching point pairs other than the first initial matching point pair among the plurality of initial matching point pairs and the epipolar line; Step 3: determine whether the distance is greater than a threshold distance. When the distance is less than the threshold distance, the second initial matching point pair is an inner point. When the distance is greater than or equal to the threshold distance, the second initial matching point pair is an outer point. Step 4: determine whether the ratio of the number of inliers to the number of the multiple initial matching point pairs is greater than a threshold ratio and whether the number of iterations is greater than a maximum number of iterations. If the ratio of the number of inliers to the number of the multiple initial matching point pairs is greater than the threshold ratio, or the number of iterations is greater than the maximum number of iterations, the initial basic matrix is ​​the target basic matrix. If the ratio of the number of inliers to the number of the multiple initial matching point pairs is less than or equal to the threshold ratio, and the number of iterations is less than or equal to the maximum number of iterations, repeat steps 1 to 4. The determining of the first projection matrix and the second projection matrix based on the center point of the first image, the first pole and the second pole comprises: Performing homogeneous transformation and simplification on the center point of the first image to obtain an initial translation transformation matrix; Performing rotation mapping on the first image and the second image to obtain a first rotation matrix and a second rotation matrix respectively; Project the first pole and the second pole to a point at infinity in the horizontal direction, and correspondingly obtain a first standard projection transformation matrix and a second standard projection transformation matrix; Performing translation transformation on the first image and the second image to obtain a target translation transformation matrix; Determine the first projection matrix based on the initial translation transformation matrix, the first rotation matrix, the first standard projection transformation matrix, and the target translation transformation matrix; The second projection matrix is ​​determined based on the initial translation transformation matrix, the second rotation matrix, and the second standard projection transformation matrix.

2. The epipolar correction method according to claim 1, characterized in that: The first Gaussian pyramid includes a plurality of first image layers, and the dividing the first Gaussian pyramid based on a preset division rule to obtain a plurality of first sub-image blocks includes: Dividing each of the plurality of first image layers to obtain a plurality of first image regions; Each of the multiple first image areas is divided to obtain multiple first sub-image blocks.

3. The epipolar line correction method according to claim 2, characterized in that: The multiple first image layers include a current first image layer and a first adjacent image layer and a second adjacent image layer adjacent to the current first image layer, and the multiple first sub-image blocks include a current first sub-image block and a plurality of adjacent first sub-image blocks adjacent to the current first sub-image block; The determining the plurality of first feature points according to the local sliding window and the plurality of first sub-image blocks comprises: Acquire the original image grayscale of the current first sub-image block in the local sliding window; Moving the local sliding window to traverse the current first sub-image block, and acquiring the current image grayscale of the current first sub-image block in the moved local sliding window; Determine a grayscale difference sum based on the original image grayscale and the current image grayscale; When the grayscale difference is greater than or equal to the difference threshold, taking the pixel with the largest grayscale value in the current first sub-image block as a key point; Determine a first adjacent point in the first adjacent image layer corresponding to the key point and a second adjacent point in the second adjacent image layer corresponding to the key point; When the grayscale value of the key point is greater than the grayscale value of the first adjacent point, the grayscale value of the key point is greater than the grayscale value of the second adjacent point, and the grayscale value of the key point is greater than the grayscale value of each pixel in the multiple adjacent first sub-image blocks, the key point is the first feature point.

4. The epipolar correction method according to claim 1, characterized in that: The epipolar line correction method further comprises: Optimize the first projection matrix and the second projection matrix based on the optimization model to obtain a first optimized projection matrix and a second optimized projection matrix; The performing projection transformation on the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain a corrected image is specifically: The first feature points and the second feature points are projected and transformed based on the first optimized projection matrix and the second optimized projection matrix to obtain a corrected image.

5. The epipolar line correction method according to claim 4, characterized in that: The initial matching point pair includes a first matching point and a second matching point, and the optimization model is: Where S is the optimization target; H0 is the first projection matrix; H1 is the second projection matrix; is the coordinate value of the first matching point; is the coordinate value of the second matching point; N is the number of initial matching point pairs.

6. A polar line correction device, characterized in that: include: An image acquisition unit, configured to acquire a first image and a second image based on a binocular vision system; a feature point determination unit, configured to determine a plurality of first feature points in the first image and a plurality of second feature points in the second image based on grayscale values ​​of image pixels; A matching point determination unit, used for matching the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs; A basic matrix determination unit, used to determine a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and the plurality of initial matching point pairs; a projection matrix determining unit, configured to determine a first pole and a second pole based on the target base matrix and the epipolar constraint, and to determine a first projection matrix and a second projection matrix based on a center point of the first image, the first pole and the second pole; an image correction unit, configured to perform a projection transformation on the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain a corrected image; The determining of a plurality of first feature points in the first image and a plurality of second feature points in the second image based on the grayscale values ​​of the image pixels comprises: Performing smoothing processing on the first image and the second image to obtain a first smoothed image and a second smoothed image respectively; Performing multi-scale decomposition on the first smoothed image and the second smoothed image to obtain a first Gaussian pyramid and a second Gaussian pyramid respectively; Dividing the first Gaussian pyramid and the second Gaussian pyramid based on a preset division rule, and correspondingly obtaining a plurality of first sub-image blocks and a plurality of second sub-image blocks; constructing a local sliding window according to the first sub-image block or the second sub-image block, and determining the plurality of first feature points according to the local sliding window and the plurality of first sub-image blocks, and determining the plurality of second feature points according to the local sliding window and the plurality of second sub-image blocks; The step of matching the plurality of first feature points with the plurality of second feature points to obtain a plurality of initial matching point pairs includes: Determine the Euclidean distance between each first feature point in the plurality of first feature points and each second feature point in the plurality of second feature points; Determine whether the Euclidean distance is greater than a preset distance, and when the Euclidean distance is greater than the preset distance, the first feature point and the second feature point are the initial matching point pair; The determining of the target basic matrix based on the normalized eight-point algorithm, the random consistency algorithm and the multiple initial matching point pairs includes: Step 1: randomly determine eight first initial matching point pairs from the multiple initial matching point pairs, and determine an initial basic matrix and epipolar lines according to the eight first initial matching point pairs; Step 2: determining the distances between a plurality of second initial matching point pairs other than the first initial matching point pair among the plurality of initial matching point pairs and the epipolar line; Step 3: determine whether the distance is greater than a threshold distance. When the distance is less than the threshold distance, the second initial matching point pair is an inner point. When the distance is greater than or equal to the threshold distance, the second initial matching point pair is an outer point. Step 4: determine whether the ratio of the number of inliers to the number of the multiple initial matching point pairs is greater than a threshold ratio and whether the number of iterations is greater than a maximum number of iterations. If the ratio of the number of inliers to the number of the multiple initial matching point pairs is greater than the threshold ratio, or the number of iterations is greater than the maximum number of iterations, the initial basic matrix is ​​the target basic matrix. If the ratio of the number of inliers to the number of the multiple initial matching point pairs is less than or equal to the threshold ratio, and the number of iterations is less than or equal to the maximum number of iterations, repeat steps 1 to 4. The determining of the first projection matrix and the second projection matrix based on the center point of the first image, the first pole and the second pole comprises: Performing homogeneous transformation and simplification on the center point of the first image to obtain an initial translation transformation matrix; Performing rotation mapping on the first image and the second image to obtain a first rotation matrix and a second rotation matrix respectively; Project the first pole and the second pole to a point at infinity in the horizontal direction, and correspondingly obtain a first standard projection transformation matrix and a second standard projection transformation matrix; Performing translation transformation on the first image and the second image to obtain a target translation transformation matrix; Determine the first projection matrix based on the initial translation transformation matrix, the first rotation matrix, the first standard projection transformation matrix, and the target translation transformation matrix; The second projection matrix is ​​determined based on the initial translation transformation matrix, the second rotation matrix, and the second standard projection transformation matrix.

Images