Asymmetric binocular stereoscopic vision system epipolar alignment method and device and storage medium

Abstract

The invention discloses an epipolar alignment method and device for an asymmetric binocular stereoscopic vision system and a storage medium, and the method comprises the steps: carrying out the calibration of two eyes in the asymmetric binocular stereoscopic vision system, and obtaining key parameters; the key parameters comprise internal parameters and external parameters; internal parameters are adjusted, and binocular calibration is carried out again and key parameters are obtained; based on the rotation matrixes in the reacquired external parameters, rotation quantity sharing is adopted to obtain rotation matrixes of the left image and the right image, then directional rotation is carried out on translation vectors in the reacquired external parameters, final rotation matrixes of the left image and the right image are obtained, and perspective transformation matrixes are calculated respectively; performing perspective transformation on the left and right images based on the perspective transformation matrix; and based on the left and right images after perspective transformation, searching a maximum common inscribed regular rectangle by adopting a central epipolar circular scanning method, further cutting the left and right images after perspective transformation, and then matching by adopting one-dimensional parallax. According to the method, the YZ parallax of the asymmetric binocular can be thoroughly eliminated, and more effective pictures can be reserved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an epipolar alignment method, apparatus, terminal device, and computer-readable storage medium for an asymmetric binocular stereo vision system. Background Technology

[0002] In binocular stereo vision technology, traditional symmetrical binocular systems employ two cameras with identical optical performance (focal length, pixel center, distortion coefficient) and electrical performance (resolution, frame rate), and are installed strictly according to a "collinear and coplanar" configuration (with virtually no parallax in the Y and Z directions). (See reference...) Figure 1 This system, after eliminating residual disparity through an epipolar alignment algorithm, only requires one-dimensional (X-direction) disparity matching to efficiently obtain dense, high-precision depth maps. (See reference for more information.) Figure 2 It is widely used in fields such as industrial inspection and robot navigation.

[0003] However, with the popularization of mobile devices, mobile phones and other devices are often equipped with multiple cameras, and their optical parameters and sensor performance are different. As a result, there are significant differences between the cameras in the asymmetric binocular system: for example, the combination of ultra-wide-angle and ordinary wide-angle lenses has different focal lengths, pixel centers and distortion coefficients, and they cannot meet the strict requirements of collinearity and coplanarity when installed, resulting in obvious parallax in the Y direction (vertical offset) and Z direction (front and back offset). Figure 3 The blue box (large box) is the lower left ultra-wide-angle lens, and the red box (small box) is the upper right regular wide-angle lens. It can be seen that the focal length (field of view) and position are severely asymmetrical.

[0004] Despite the advantages of mobile phones and other devices, such as small size, high integration, and built-in high-performance processors and power supplies, depth estimation still faces technical bottlenecks. The core problem lies in the inability of traditional epipolar alignment algorithms to effectively handle large-scale YZ parallax in asymmetric binocular vision: these algorithms can only correct small-range parallax, and cannot completely eliminate the large-scale parallax caused by tilted or offset installation of mobile phone cameras. After correction, parallax in the Y direction still remains, thus hindering subsequent one-dimensional parallax matching and accurate acquisition of depth information.

[0005] Therefore, existing technologies have not yet solved the problem of efficient and high-precision epipolar alignment in asymmetric binocular systems, which limits the application potential of miniaturized binocular devices such as mobile phones in scenarios that rely on depth perception, such as AR measurement and 3D reconstruction. Summary of the Invention

[0006] To address the shortcomings of the prior art, this invention provides an epipolar alignment method, apparatus, terminal device, and computer-readable storage medium for an asymmetric binocular stereo vision system. This method can completely eliminate YZ parallax in asymmetric binocular vision, retain more effective images, and is compatible with efficient one-dimensional matching epipolar alignment methods.

[0007] The first objective of this invention is to provide a method for epipolar alignment in an asymmetric binocular stereo vision system.

[0008] The second objective of this invention is to provide an epipolar alignment device for an asymmetric binocular stereo vision system.

[0009] The third objective of this invention is to provide a terminal device.

[0010] A fourth objective of this invention is to provide a computer-readable storage medium.

[0011] The first objective of this invention can be achieved by adopting the following technical solution: An epipolar alignment method for an asymmetric binocular stereo vision system, the method comprising: Calibrate the binoculars in an asymmetric binocular stereo vision system to obtain key parameters, including intrinsic and extrinsic parameters. Adjust the intrinsic parameters to align the binocular focal length, then recalibrate the binoculars and obtain the key parameters; Based on the rotation matrix in the reacquired extrinsic parameters, the rotation amount is amortized to obtain the rotation matrices of the left and right images; the left and right images are the images captured by the left and right eyes in binocular vision, respectively. Based on the rotation matrices of the left and right images, the translation vectors in the newly acquired extrinsic parameters are oriented and rotated to obtain the final rotation matrices of the left and right images; Based on the final rotation matrices and intrinsic parameters of the left and right images, the perspective transformation matrices are calculated respectively; based on the perspective transformation matrices, perspective transformation is performed on the left and right images to achieve image correction. Based on the left and right images after perspective transformation, the epipolar circular scanning method is used to search for the longest common inscribed rectangle; the longest common inscribed rectangle is then used to crop the left and right images after perspective transformation respectively. One-dimensional disparity matching is used on the cropped left and right images to obtain depth information.

[0012] Furthermore, the intrinsic parameters include left-eye intrinsic parameters and right-eye intrinsic parameters. The left-eye intrinsic parameters include the focal lengths flx and fly in the X and Y directions and the pixel center coordinates clx and cly. The right-eye intrinsic parameters include the focal lengths frx and fry in the X and Y directions and the pixel center coordinates crx and cry.

[0013] Furthermore, the search for the greatest common inscribed rectangle using the epipolar circular scanning method based on the left and right images after perspective transformation includes: Using the pixel center coordinates in the left eye intrinsic parameters as the center, generate several straight lines passing through the center at different angles, and record the intersection points of the straight lines with the effective areas of the left and right images after perspective transformation; different angles include 0~180°; Select the vertices that can form a regular rectangle from all intersections, and find the one with the largest area as the final target common inscribed regular rectangle.

[0014] Furthermore, the calculation of the perspective transformation matrix based on the rotation matrix and intrinsic parameters of the final left and right images includes: The intrinsic parameter matrix Kl of the left eye is composed of the intrinsic parameters of the left eye, and Kl serves as the reference intrinsic parameter matrix K of the corrected image. Calculate the perspective transformation matrix of the left image according to Hl=K×RlRrowAlign×inv(Kl); RlRrowAlign is the rotation matrix of the final left image, and inv(Kl) is the inverse matrix of Kl; The perspective transformation matrix of the right image is calculated according to Hr=K×RrRrowAlign×inv(Kr); RrRrowAlign is the rotation matrix of the final right image; Kr is the intrinsic parameter matrix of the right image, which is composed of the intrinsic parameters of the right image.

[0015] Furthermore, the rotation matrices based on the left and right images are used to perform directional rotation on the translation vectors in the re-acquired extrinsic parameters to obtain the final rotation matrices for the left and right images, including: Applying the rotation matrix Rr of the right image to the translation vector T yields the translation vector t; Calculate the cross product of the translation vector t and the unit vector of the X-axis to obtain the rotation axis; Divide the axis of rotation by its length to obtain the normalized axis of rotation; The rotation angle is obtained by calculating the angle between the translation vector t and the unit vector of the X-axis using the vector dot product formula. Multiply the rotation angle by the normalized rotation axis, and then convert it into the rotation matrix RrowAlign through the Rodrigues transformation; Calculate the final rotation matrix of the left image based on RlRrowAlign=RrowAlign@Rl; Rl is the rotation matrix of the left image. Calculate the final rotation matrix of the right figure based on RrRrowAlign=RrowAlign@Rr.

[0016] Furthermore, the rotation matrices of the left and right images are obtained by amortizing the rotation amount based on the reacquired extrinsic parameters, including: Transform the rotation matrix R into a rotation vector r using the Rodrigues transformation; Divide the rotation vector r by -2, and then transform it using the Rodrigues transformation to obtain the rotation matrix of the right image; Use the transpose of the rotation matrix of the right image as the rotation matrix of the left image.

[0017] Furthermore, the adjustment of the intrinsic parameters involves adjusting the focal lengths in the X and Y directions of the left eye's intrinsic parameters to be completely consistent with the focal lengths in the X and Y directions of the right eye's intrinsic parameters.

[0018] The second objective of this invention can be achieved by adopting the following technical solution: An epipolar alignment device for an asymmetric binocular stereo vision system, the system comprising: The parameter acquisition module is used to calibrate the binoculars in an asymmetric binocular stereo vision system and acquire key parameters, including intrinsic and extrinsic parameters. The recalibration module is used to adjust the intrinsic parameters to align the binocular focal length, then recalibrate the binoculars and obtain key parameters; The rotation amortization module is used to amortize the rotation matrix in the reacquired extrinsic parameters to obtain the rotation matrices of the left and right images; the left and right images are the images captured by the left and right eyes in the binoculars, respectively. The orientation rotation module is used to perform orientation rotation on the translation vector in the reacquired extrinsic parameters based on the rotation matrices of the left and right images, so as to obtain the final rotation matrices of the left and right images. The correction module is used to calculate the perspective transformation matrix based on the final rotation matrix and intrinsic parameters of the left and right images respectively; and to perform perspective transformation on the left and right images based on the perspective transformation matrix to achieve image correction. The search module is used to search for the greatest common inscribed rectangle based on the left and right images after perspective transformation using the epipolar circular scanning method; and then uses the greatest common inscribed rectangle to crop the left and right images after perspective transformation respectively. The matching module is used to perform one-dimensional disparity matching on the cropped left and right images to obtain depth information.

[0019] The third objective of this invention can be achieved by adopting the following technical solution: A terminal device includes a processor and a memory for storing a processor-executable program. When the processor executes the program stored in the memory, it implements the epipolar alignment method of the asymmetric binocular stereo vision system described above.

[0020] The fourth objective of this invention can be achieved by adopting the following technical solution: A computer-readable storage medium storing a program that, when executed by a processor, implements the epipolar alignment method of the asymmetric binocular stereo vision system described above.

[0021] The present invention has the following advantages over the prior art: (1) Completely eliminate YZ parallax and be compatible with efficient one-dimensional matching: By directional rotation of translation vector, the YZ parallax of asymmetric binoculars is reduced to near zero, so that traditional one-dimensional matching algorithms (such as SGBM) can be used afterward. The matching efficiency is 3 to 5 times higher than that of two-dimensional matching, and dense point clouds can be generated. (2) Reduce image loss and retain more effective information: By using a rotation amount distribution strategy, the left and right images each bear half of the rotation amount, avoiding edge cropping caused by excessive rotation of a single image, and improving the effective visual information retention rate by more than 40%. (3) Improve the effective matching area and expand the depth coverage: The search is performed using the maximum common inscribed rectangle. Compared with the traditional square clipping, the effective matching area is increased by at least 20%, especially on binocular devices with low asymmetry, the increase can reach 50%. (4) Wide range of applications and low hardware cost: It is compatible with asymmetric binocular devices such as mobile phones and tablets, and does not require dedicated binocular hardware (such as professional binocular cameras from Orbbec and Intel). It can directly utilize the existing cameras and processors of mobile devices such as mobile phones and tablets. Since the cost of binocular hardware has been reduced on the original basis, the hardware cost has been reduced by 60% to 80%. (5) High accuracy and strong practicality: At a distance of 30cm, the depth estimation error can be controlled to about 0.1mm, and the algorithm can be ported to mobile systems such as Android, supporting localized processing and meeting the needs of AR measurement, small-scale 3D reconstruction and other scenarios. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a traditional symmetrical binocular system in the background art of this invention; Figure 2 This is a schematic diagram of an image captured by a traditional symmetrical binocular system in the background art of this invention; wherein, the blue border represents the image captured by the left eye, and the red border represents the image captured by the right eye; Figure 3 This is a schematic diagram of an image captured by an asymmetric binocular system in the background art of this invention; Figure 4 This is a flowchart of the epipolar alignment method for the asymmetric binocular stereo vision system in Embodiment 1 of the present invention; Figure 5These are the left and right images of the corrected Z-parallax in Embodiment 1 of the present invention; Figure 6 These are the left and right images of the corrected Z-parallax and corrected Y-parallax in Embodiment 1 of the present invention; Figure 7 This is an overlay of the left and right views after perspective transformation according to Embodiment 1 of the present invention; Figure 8 This refers to the area of ​​the traditional inscribed circle and then inscribed square cutting method of Embodiment 1 of the present invention; Figure 9 The maximum common inscribed rectangle after trimming in Embodiment 1 of the present invention; Figure 10 This is a structural block diagram of the epipolar alignment device of the asymmetric binocular stereo vision system according to Embodiment 2 of the present invention; Figure 11 This is a structural block diagram of the terminal device according to Embodiment 3 of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be understood that the specific embodiments described are merely used to explain this application and are not intended to limit this application.

[0025] Example 1:

[0026] like Figure 4 As shown, this embodiment provides an epipolar alignment method for an asymmetric binocular stereo vision system, including the following steps: S401. Calibrate the binoculars in the asymmetric binocular stereo vision system and obtain key parameters; key parameters include intrinsic and extrinsic parameters.

[0027] For asymmetric binocular devices such as mobile phones, calibration is performed to obtain key parameters of the two cameras (left camera and right camera), including intrinsic parameters of the left camera (left eye), intrinsic parameters of the right camera (right eye), and extrinsic parameters. The intrinsic parameters Kl of the left camera include the focal length (flx, fly) in the X / Y direction and the pixel center coordinates (clx, cly). The intrinsic parameters Kr of the right camera include the focal length (frx, fry) in the X / Y direction and the pixel center coordinates (crx, cry). The extrinsic parameters include the rotation matrix R and translation vector T of the right camera relative to the left camera.

[0028] Specifically, the rotation matrix R is a 3×3 matrix that describes the relative rotation relationship between the two cameras; the translation vector T is a 3×1 vector that describes the relative positional offset between the two, such as Tx, Ty, and Tz corresponding to the offsets in the X, Y, and Z directions, respectively.

[0029] The calibration process can use a common checkerboard calibration board: place the calibration board within the camera's field of view, take 10 to 20 sets of images from different angles and distances, and calculate the above parameters using a calibration algorithm (such as Zhang Zhengyou's calibration method) to ensure that the parameter error is controlled within 0.5 pixels.

[0030] S402. Adjust the internal parameters to align the binocular focal length, then recalibrate the binoculars and obtain the key parameters.

[0031] By adjusting the intrinsic parameters to align the focal lengths of both eyes, most of the Z-parallax can be eliminated. (See also...) Figure 5 .

[0032] Force the focal length of the left camera with a larger field of view to be reduced to be the same as that of the right camera: the focal length flx and fly in the X / Y direction of the intrinsic parameters Kl of the left camera and the focal length frx and fry in the X / Y direction of the intrinsic parameters Kr of the right camera are completely the same; then re-execute step S401 to recalibrate and obtain the binocular parameters.

[0033] S403. Based on the rotation matrix in the reacquired extrinsic parameters, the rotation amount is amortized to obtain the rotation matrices of the left and right images.

[0034] By distributing the rotation amount, image loss can be reduced.

[0035] The image captured by the left camera is referred to as the left image Img_l, and the image captured by the right camera is referred to as the right image Img_r.

[0036] Traditional epipolar alignment only rotates the image captured by the right camera (rotating the right image to the coordinate system of the left image), which can easily lead to the cropping of content at the edges of the right image.

[0037] This embodiment employs a rotation amortization strategy, splitting the rotation amount of the rotation matrix R into two parts, rotating the left and right images by half each. Specifically, this includes: The rotation matrix R is transformed into a rotation vector r using the Rodrigues transformation, that is, a 3×3 rotation matrix is ​​transformed into a 3×1 rotation vector. Vector r represents the rotation direction and angle of the right camera relative to the left camera.

[0038] To calculate the rotation matrix Rr of the right image: divide the rotation vector r by -2, which halves the rotation angle and reverses the direction, and then transform it through the Rodrigues transformation to obtain the 3×3 rotation matrix Rr; Calculate the rotation matrix Rl of the left image: Rl is the transpose of Rr to ensure that the rotation amounts of the left and right images are equal in magnitude and opposite in direction, thus achieving bidirectional alignment.

[0039] In this way, the rotation angles of the left and right images are only half that of the traditional method, which greatly reduces the cropping loss at the image edges and retains more effective visual information.

[0040] S404. Based on the rotation matrices of the left and right images, perform orientational rotation on the translation vectors in the newly acquired extrinsic parameters to obtain the final rotation matrices of the left and right images.

[0041] Oriented rotation and translation vectors are used to eliminate Y-parallax and the remaining Z-parallax.

[0042] Asymmetric binocular translation vector T typically contains offsets in the Y and Z directions, with Ty and Tz ≠ 0, leading to YZ parallax. This step eliminates YZ parallax completely by rotating the translation vector T to retain only the X-direction offset, thus ensuring Ty ≈ 0 and Tz ≈ 0. Specifically, this includes: (1) Synchronous rotation of translation vector: Apply the rotation matrix Rr of the right image obtained in step S403 to the translation vector T to obtain a new translation vector t, i.e., t=Rr×T, to ensure that the translation vector is consistent with the coordinate system of the rotated image; (2) Calculate the rotation axis: With the X-axis unit vector as the target direction, calculate the cross product of the translation vector t and the X-axis unit vector to obtain the rotation axis; the rotation axis is perpendicular to the plane formed by t and the X-axis to ensure that t can point to the X-axis after rotation; In this embodiment, the X-axis unit vector is [1,0,0], representing the horizontal direction.

[0043] (3) Calculate the normalized rotation axis: Divide the calculated rotation axis by the length of the rotation axis to obtain the normalized rotation axis, so as to avoid the error in the calculation of the rotation angle; (4) Calculate the rotation angle: Calculate the angle between the translation vector t and the X-axis unit vector using the vector dot product formula, i.e., angle = arccos((t・X-axis unit vector) / (|t|×|X-axis unit vector|)), which is the angle that the translation vector needs to rotate. (5) Construct the rotation matrix RrowAlign: multiply the rotation angle by the normalized rotation axis, and then convert it into a 3×3 rotation matrix RrowAlign through Rodrigues transformation; (6) Oriented rotation and translation vector: Apply RrowAlign to the translation vector t to obtain the final translation vector T'; at this time, the Ty and Tz components of T' are close to 0, only Tx' in the X direction is retained, and the YZ parallax is completely eliminated; (7) Calculate the final rotation matrix: RlRrowAlign=RrowAlign@Rl is the final rotation matrix of the left figure, and RrRrowAlign=RrowAlign@Rr is the final rotation matrix of the right figure.

[0044] S405. Based on the rotation matrices and intrinsic parameters of the left and right images, calculate the perspective transformation matrix respectively; based on the perspective transformation matrix, perform perspective transformation on the left and right images to achieve image correction.

[0045] Image correction is achieved by calculating the perspective transformation matrix.

[0046] To apply the rotation effect of the left and right images to the actual pixels, it is necessary to calculate the perspective transformation matrix of both images, specifically including: (1) Determine the reference intrinsic parameters: The intrinsic parameter matrix Kl of the left camera is composed of flx, fly, clx, and cly. Kl serves as the reference intrinsic parameter matrix K of the corrected image. Specifically, the intrinsic parameter matrix Kl is a 3x3 matrix: [[fx,0,cx],[0,fy,cy],[0,0,1]].

[0047] (2) Perspective transformation matrix of the left image: Calculate the perspective transformation matrix of the left image according to Hl=K×RlRrowAlign×inv(Kl); inv(Kl) is the inverse matrix of Kl, which is to reverse the pixel coordinates to 3D relative coordinates; and then rotate through Rl and project back to 2D pixel coordinates through K. (3) Perspective transformation matrix of the right image: The perspective transformation matrix of the right image is calculated according to Hr=K×RrRrowAlign×inv(Kr); Kr is the intrinsic parameter matrix of the right camera, which is composed of frx, fry, crx, and cry; (4) Image perspective transformation: Input the transpose matrices Hl.T and Hr.T of the corresponding perspective transformation matrices into the left and right images respectively to obtain the corrected left image Img_l' and right image Img_r': At this time, the lines connecting the corresponding feature points of the two images are completely horizontal (no YZ parallax), and only X-direction parallax exists. Hl.T is the transpose matrix of Hl, and Hr.T is the transpose matrix of Hr.

[0048] refer to Figure 6 After correcting the Z-parallax and then the Y-parallax, the lines connecting each content in the left and right images are horizontal, with no vertical parallax.

[0049] S406. Based on the left and right images after perspective transformation, the maximum common inscribed rectangle is searched using the epipolar circular scanning method; the maximum common inscribed rectangle is then used to crop the left and right images after perspective transformation.

[0050] By employing a search with the maximum common inscribed rectangle, black border interference is reduced.

[0051] After perspective transformation, due to differences in asymmetrical binocular field of view (Z-parallax) and different installation positions, perspective transformation with epipolar alignment will result in numerous meaningless black edges (the specific color depends on the user). Black borders will appear at the image edges (invalid areas caused by rotation), such as... Figure 7 If used directly for disparity calculation and matching, it will lead to mismatches. Therefore, it is necessary to find a common inscribed regular rectangle (a rectangle with a rotation angle of 0) in the left and right images, and use this as the image cropping area to crop the same area in the left and right images, ensuring that there are no meaningless black border areas in the cropped left and right images, that is, that they are completely within the effective area of ​​the left and right images, and ensuring that the area of ​​this rectangle is maximized, thereby ensuring that the area that can be matched by disparity is maximized.

[0052] The most direct enumeration method involves enumerating all combinations of the rectangle's four values ​​(top-left corner x, top-left corner y, width, height) and then checking if the rectangle is simultaneously within the valid areas of the left and right images. The computational complexity is O(n^4), where n is the total number of data points. Using the traditional method of inscribed circle followed by inscribed square for cropping, the complexity is O(1), but the area is much smaller (see reference). Figure 8 The lowest returns are observed when the difference between the left and right cameras is small.

[0053] This embodiment searches for the largest common inscribed rectangle using the center epipolar circular scanning method, with a computational complexity of only O(n), significantly improving computational efficiency. The area obtained is at least 20% larger than the traditional method of inscribed circle followed by inscribed square trimming. Specifically, it includes: (1) Determine the search benchmark: Since the effective area of ​​the left and right images is close to a rotated rectangle, the center of the common rectangle with the largest area is likely to appear near the mean center of the image, so as to ensure that the area is large enough. Assuming that the mean optical center of the corrected left and right images is taken as the center of the rectangle, one of the diagonals of the target rectangle passes through the mean optical center of the left and right images and intersects with the image edge of the overlapping left and right images; (2) Circular scanning verification: With (clx,cly) as the center, generate several straight lines with different angles (0-180°) passing through the center of the circle, and record the intersection points of the straight lines with the effective areas (non-black border areas) of the corrected left and right images; (3) Determine the range of the rectangle: Select the vertices that can form a regular rectangle from all intersections, ensure that the vertices of the rectangle are located within the effective areas of the left and right images at the same time, and find the one with the largest area, which is regarded as the final target common inscribed regular rectangle; (4) Image cropping: The corrected left and right images are cropped according to the common inscribed rectangle of the final target to obtain the final left-right aligned images Img_l'' and Img_r''. (See reference) Figure 9The thick black bounding box is a maximum common inscribed rectangle with no black border, and its effective matching area is at least 20% larger than the traditional inscribed circle and then inscribed square clipping method (thin-line circle and thin-line square). The benefit is even greater when the YZ parallax of the left and right cameras is small. Even in the most extreme and worst-case 45-degree tilt condition, the area of ​​the thick black bounding rectangle is still 20% larger than the thin black bounding square of the traditional algorithm. S407. One-dimensional disparity matching is used on the cropped left and right images to obtain depth information.

[0054] The baseline intrinsic parameter matrix K depends on whether the main camera is a left-eye (left camera) or a right-eye (right camera); Kl is used for the left-eye and Kr for the right-eye. Based on the baseline intrinsic parameter matrix K and the final translation vector T', a reprojection matrix Q is established, which is a current technique. The reprojection matrix Q is used to convert the subsequently obtained disparity map into a 3D point cloud.

[0055] The aligned images Img_l'' and Img_r'' are directly matched using traditional, efficient one-dimensional (X-direction) binocular matching algorithms (such as BM and SGBM models) to output a dense, high-precision disparity map. The disparity map is then used to calculate the point cloud through the reprojection matrix Q, thereby obtaining depth information.

[0056] Among them, the one-dimensional (X-direction) binocular matching algorithm (one-dimensional disparity matching) is an existing method.

[0057] To further improve the accuracy, density, and development efficiency of disparity calculation, this embodiment employs a deep learning-driven binocular disparity estimation model to replace traditional BM and SGBM algorithms, such as advanced AI models like Sintel or SceneFlow, for one-dimensional disparity matching. This type of model achieves automatic disparity optimization through end-to-end training, eliminating the need for manual adjustment of parameters such as matching window size and aggregation path weights, significantly reducing development and testing costs. Furthermore, it outperforms traditional algorithms in complex scenes with weak textures, reflections, and transparency. This AI model adopts an end-to-end process of feature extraction, cost aggregation, and disparity optimization, using a deep network to achieve stereo matching. Its main advantages include: a dual-branch backbone network collaboratively extracting monocular and stereo features; a side-tuned adapter enhancing weak texture matching; dual correlation calculation fusing local and global information to construct a 3D cost volume; an attention module combining axial convolution and Transformer for cost processing; and optimization with Soft-Argmin and ConvGRU, with weighted L1 loss ensuring matching performance. The model adopts a modular cascaded design, encompassing core modules such as the feature backbone and side-tuned adapter. In terms of training and deployment, it is trained on a synthetic dataset of millions of records, exhibiting strong generalization capabilities, zero-parameter deployment, and adaptability to mobile devices. Its performance is outstanding; compared to the traditional SGBM algorithm, it shows significant improvements in accuracy, density, and robustness in the Middlebury benchmark, with clear advantages in areas with no texture, reflections, and transparent objects. At a distance of 30cm, the depth estimation error can be controlled to around 0.1mm.

[0058] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium.

[0059] It should be noted that although the method operations of the above embodiments are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the order of execution of the described steps may be changed. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. Example 2:

[0060] like Figure 10 As shown, this embodiment provides an epipolar alignment device for an asymmetric binocular stereo vision system. The device includes a parameter acquisition module 1001, a recalibration module 1002, a rotation amount amortization module 1003, a orientation rotation module 1004, a correction module 1005, a search module 1006, and a matching module 1007, wherein: The parameter acquisition module 1001 is used to calibrate the binoculars in an asymmetric binocular stereo vision system and acquire key parameters, including intrinsic and extrinsic parameters. The recalibration module 1002 is used to adjust the intrinsic parameters to align the binocular focal length, then recalibrate the binoculars and obtain key parameters. The rotation amount amortization module 1003 is used to obtain the rotation matrices of the left and right images by amortizing the rotation matrix in the reacquired extrinsic parameters; the left and right images are the images captured by the left and right eyes in the binoculars, respectively. The orientation rotation module 1004 is used to perform orientation rotation on the translation vector in the reacquired extrinsic parameters based on the rotation matrices of the left and right images, so as to obtain the final rotation matrices of the left and right images. The correction module 1005 is used to calculate the perspective transformation matrix based on the final rotation matrix and intrinsic parameters of the left and right images respectively; and to perform perspective transformation on the left and right images based on the perspective transformation matrix to achieve image correction. The search module 1006 is used to search for the greatest common inscribed rectangle based on the left and right images after perspective transformation using the epipolar circular scanning method; and to crop the left and right images after perspective transformation using the greatest common inscribed rectangle. The matching module 1007 is used to perform one-dimensional disparity matching on the cropped left and right images to obtain depth information.

[0061] The specific implementation of each module in this embodiment can be found in Embodiment 1 above, and will not be repeated here. It should be noted that the device provided in this embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above. Example 3:

[0062] This embodiment provides a terminal device, which can be a computer, such as... Figure 11 As shown, the system is connected via a system bus 1101 to a processor 1102, a memory, an input device 1103, a display 1104, and a network interface 1105. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 1106 and internal memory 1107. The non-volatile storage medium 1106 stores the operating system, computer programs, and a database. The internal memory 1107 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the processor 1102 executes the computer program stored in the memory, it implements the epipolar alignment method of the asymmetric binocular stereo vision system described in Embodiment 1, as follows: Calibrate the binoculars in an asymmetric binocular stereo vision system to obtain key parameters, including intrinsic and extrinsic parameters. Adjust the intrinsic parameters to align the binocular focal length, then recalibrate the binoculars and obtain the key parameters; Based on the rotation matrix in the reacquired extrinsic parameters, the rotation amount is amortized to obtain the rotation matrices of the left and right images; the left and right images are the images captured by the left and right eyes in binocular vision, respectively. Based on the rotation matrices of the left and right images, the translation vectors in the newly acquired extrinsic parameters are oriented and rotated to obtain the final rotation matrices of the left and right images; Based on the final rotation matrices and intrinsic parameters of the left and right images, the perspective transformation matrices are calculated respectively; based on the perspective transformation matrices, perspective transformation is performed on the left and right images to achieve image correction. Based on the left and right images after perspective transformation, the epipolar circular scanning method is used to search for the longest common inscribed rectangle; the longest common inscribed rectangle is then used to crop the left and right images after perspective transformation respectively. One-dimensional disparity matching is used on the cropped left and right images to obtain depth information. Example 4:

[0063] This embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the epipolar alignment method of the asymmetric binocular stereo vision system described in Embodiment 1 above, as follows: Calibrate the binoculars in an asymmetric binocular stereo vision system to obtain key parameters, including intrinsic and extrinsic parameters. Adjust the intrinsic parameters to align the binocular focal length, then recalibrate the binoculars and obtain the key parameters; Based on the rotation matrix in the reacquired extrinsic parameters, the rotation amount is amortized to obtain the rotation matrices of the left and right images; the left and right images are the images captured by the left and right eyes in binocular vision, respectively. Based on the rotation matrices of the left and right images, the translation vectors in the newly acquired extrinsic parameters are oriented and rotated to obtain the final rotation matrices of the left and right images; Based on the final rotation matrices and intrinsic parameters of the left and right images, the perspective transformation matrices are calculated respectively; based on the perspective transformation matrices, perspective transformation is performed on the left and right images to achieve image correction. Based on the left and right images after perspective transformation, the epipolar circular scanning method is used to search for the longest common inscribed rectangle; the longest common inscribed rectangle is then used to crop the left and right images after perspective transformation respectively. One-dimensional disparity matching is used on the cropped left and right images to obtain depth information.

[0064] It should be noted that the computer-readable storage medium in this embodiment can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0065] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A method for epipolar alignment in an asymmetric binocular stereo vision system, characterized in that, The method includes: Calibrate the binoculars in an asymmetric binocular stereo vision system to obtain key parameters, including intrinsic and extrinsic parameters. Adjust the intrinsic parameters to align the binocular focal length, then recalibrate the binoculars and obtain the key parameters; Based on the rotation matrix in the reacquired extrinsic parameters, the rotation amount is amortized to obtain the rotation matrices of the left and right images; the left and right images are the images captured by the left and right eyes in binocular vision, respectively. Based on the rotation matrices of the left and right images, the translation vectors in the newly acquired extrinsic parameters are oriented and rotated to obtain the final rotation matrices of the left and right images; Based on the final rotation matrices and intrinsic parameters of the left and right images, the perspective transformation matrices are calculated respectively; based on the perspective transformation matrices, perspective transformation is performed on the left and right images to achieve image correction. Based on the left and right images after perspective transformation, the epipolar circular scanning method is used to search for the longest common inscribed rectangle; the longest common inscribed rectangle is then used to crop the left and right images after perspective transformation respectively. One-dimensional disparity matching is used on the cropped left and right images to obtain depth information.

2. The polar alignment method according to claim 1, characterized in that, The intrinsic parameters include left-eye intrinsic parameters and right-eye intrinsic parameters. The left-eye intrinsic parameters include focal lengths in the X and Y directions and pixel center coordinates, while the right-eye intrinsic parameters include focal lengths in the X and Y directions and pixel center coordinates.

3. The polar alignment method according to claim 2, characterized in that, The left and right images based on perspective transformation are used to search for the longest common inscribed rectangle using the epipolar circular scanning method, including: Using the pixel center coordinates in the left eye intrinsic parameters as the center, generate several straight lines passing through the center at different angles, and record the intersection points of the straight lines with the effective areas of the left and right images after perspective transformation; Select the vertices that can form a regular rectangle from all intersections, and find the one with the largest area as the final target common inscribed regular rectangle.

4. The polar alignment method according to claim 2, characterized in that, The perspective transformation matrix is ​​calculated based on the rotation matrix and intrinsic parameters of the final left and right images, including: The intrinsic parameter matrix Kl of the left eye is composed of the intrinsic parameters of the left eye, and Kl serves as the reference intrinsic parameter matrix K of the corrected image. Calculate the perspective transformation matrix of the left image according to Hl=K×RlRrowAlign×inv(Kl); RlRrowAlign is the rotation matrix of the final left image, and inv(Kl) is the inverse matrix of Kl; The perspective transformation matrix of the right image is calculated according to Hr=K×RrRrowAlign×inv(Kr); RrRrowAlign is the rotation matrix of the final right image; Kr is the intrinsic parameter matrix of the right image, which is composed of the intrinsic parameters of the right image.

5. The polar alignment method according to any one of claims 1 to 4, characterized in that, The rotation matrices based on the left and right images are used to perform directional rotation on the translation vectors in the newly acquired extrinsic parameters to obtain the final rotation matrices for the left and right images, including: Applying the rotation matrix Rr of the right image to the translation vector T, we obtain the translation vector t; Calculate the cross product of the translation vector t and the unit vector of the X-axis to obtain the rotation axis; Divide the axis of rotation by its length to obtain the normalized axis of rotation; The rotation angle is obtained by calculating the angle between the translation vector t and the unit vector of the X-axis using the vector dot product formula. Multiply the rotation angle by the normalized rotation axis, and then convert it into the rotation matrix RrowAlign through the Rodrigues transformation; Calculate the final rotation matrix of the left image based on RlRrowAlign=RrowAlign@Rl; Rl is the rotation matrix of the left image. Calculate the final rotation matrix of the right figure based on RrRrowAlign=RrowAlign@Rr.

6. The polar alignment method according to any one of claims 1 to 4, characterized in that, The rotation matrix based on the reacquired extrinsic parameters is used to distribute the rotation amount to obtain the rotation matrices of the left and right images, including: Transform the rotation matrix R into a rotation vector r using the Rodrigues transformation; Divide the rotation vector r by -2, and then transform it using the Rodrigues transformation to obtain the rotation matrix of the right image; Use the transpose of the rotation matrix of the right image as the rotation matrix of the left image.

7. The polar alignment method according to any one of claims 1 to 4, characterized in that, The adjustment of the intrinsic parameters is as follows: the focal lengths in the X and Y directions of the left eye's intrinsic parameters are adjusted to be completely consistent with the focal lengths in the X and Y directions of the right eye's intrinsic parameters.

8. An epipolar alignment device for an asymmetric binocular stereo vision system, characterized in that, The system includes: The parameter acquisition module is used to calibrate the binoculars in an asymmetric binocular stereo vision system and acquire key parameters, including intrinsic and extrinsic parameters. The recalibration module is used to adjust the intrinsic parameters to align the binocular focal length, then recalibrate the binoculars and obtain key parameters; The rotation amortization module is used to amortize the rotation matrix in the reacquired extrinsic parameters to obtain the rotation matrices of the left and right images; the left and right images are the images captured by the left and right eyes in the binocular system, respectively. The orientation rotation module is used to perform orientation rotation on the translation vector in the reacquired extrinsic parameters based on the rotation matrices of the left and right images, so as to obtain the final rotation matrices of the left and right images. The correction module is used to calculate the perspective transformation matrix based on the final rotation matrix and intrinsic parameters of the left and right images respectively; and to perform perspective transformation on the left and right images based on the perspective transformation matrix to achieve image correction. The search module is used to search for the greatest common inscribed rectangle based on the left and right images after perspective transformation using the epipolar circular scanning method; and then uses the greatest common inscribed rectangle to crop the left and right images after perspective transformation respectively. The matching module is used to perform one-dimensional disparity matching on the cropped left and right images to obtain depth information.

9. A terminal device, comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the polar alignment method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the polar alignment method according to any one of claims 1 to 7.

Images