A camera calibration method, a stereo reconstruction method and an image correction method

Abstract

The present disclosure relates to a camera calibration method, comprising: acquiring a plurality of captured images, the captured images being images of a calibration board captured by the camera from different distances and / or different angles, the calibration board containing a calibration image, the calibration image containing a plurality of discrete or continuous calibration points; acquiring calibration points corresponding to each pixel of the camera in the plurality of captured images; calculating ray information of the pixel in the physical world according to the coordinates of the calibration points in the physical world, and acquiring a set of ray information corresponding to each pixel of the camera to complete the calibration of the camera. To at least partially solve the lack of universality and accuracy of traditional calibration methods.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more specifically, to a camera calibration method, a stereo reconstruction method, an image correction method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Stereo vision plays a fundamental role in numerous fields, including medical imaging, industrial inspection, 3D facial recognition, autonomous driving, animation, and virtual / augmented reality. Stereo vision aims to recover 3D geometry from pairs of 2D images captured by cameras from different viewpoints. For decades, stereo vision has been a core area of ​​computer vision and is becoming increasingly important, especially for autonomous driving.

[0003] To improve the safety of autonomous driving and enhance the quality of services based on stereo vision technology, it is crucial to improve the accuracy and efficiency of stereo vision algorithms, especially camera calibration. Summary of the Invention

[0004] Traditional stereo vision methods are based on the pinhole camera model, where all light rays converge to a single optical center. This model uses only a few parameters to describe the camera's relative position and orientation, as well as inherent distortions, including effective focal length, tilt parameters, principal center, and radial and tangential distortions. However, many real-world camera types cannot be modeled as pinhole cameras, such as mobile phone cameras, light field cameras, fisheye lenses, wide-angle lenses, or panoramic cameras. Furthermore, the distortions of optical lenses and digital sensors are highly complex due to manufacturing precision limitations. Traditional pinhole camera models cannot fully describe all distortions. In addition, traditional calibration processes are inaccurate due to calibration plate distortion and the sparsity of feature points on the plate. Theoretically, traditional camera calibration algorithms are equivalent to nonlinear optimization problems, which can lead to calibration results getting trapped in local optima. Therefore, traditional camera calibration methods struggle to meet the requirements of many modern practical applications.

[0005] In view of this, the present disclosure provides a camera calibration method, a stereo reconstruction method, an image correction method, an electronic device, and a storage medium to at least partially solve the shortcomings of the related technical solutions.

[0006] According to one embodiment of this disclosure, a camera calibration method is provided, comprising: acquiring multiple captured images, wherein the captured images are images obtained by the camera from different distances and / or different angles of a calibration board, the calibration board containing calibration images, and the calibration images containing multiple discrete or continuous calibration points; acquiring calibration points corresponding to each pixel of the camera in the multiple captured images; calculating the ray information of the corresponding pixel in the physical world based on the coordinates of the calibration points in the physical world; and acquiring a set of ray information corresponding to each pixel of the camera to complete the calibration of the camera.

[0007] According to one embodiment of this disclosure, a stereo reconstruction method is provided, comprising: capturing a first image using a first camera; acquiring a first set of ray information corresponding to each pixel of the first camera; capturing a second image using a second camera; acquiring a second set of ray information corresponding to each pixel of the second camera; and performing stereo reconstruction based on the first image, the second image, the first set, and the second set.

[0008] According to one embodiment of this disclosure, an image correction method is provided, comprising: acquiring a captured image through a camera; acquiring a set of ray information corresponding to each pixel of the camera; and generating a virtual corrected image based on the captured image and the set.

[0009] According to one embodiment of this disclosure, an electronic device is provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any of the above-mentioned camera calibration method, stereo reconstruction method, and image correction method by executing the executable instructions.

[0010] According to one embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements any of the above-mentioned camera calibration method, stereo reconstruction method, and image correction method. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0012] Figure 1 This is a schematic diagram of PCA-based linear fitting according to one embodiment of the present disclosure;

[0013] Figure 2 This is a schematic diagram of a striped pattern displayed by a digital projector or LCD according to one embodiment of the present disclosure;

[0014] Figure 3This is a schematic diagram of a striped pattern displayed by a digital projector or LCD according to one embodiment of the present disclosure;

[0015] Figure 4 This is a schematic diagram of a stripe image in a phase-shifting algorithm according to one embodiment of the present disclosure;

[0016] Figure 5 According to one embodiment of this disclosure Figure 4 A schematic diagram of the environment, modulation, and wrapping phase calculation in the striped image;

[0017] Figure 6 This is a schematic diagram comparing a physical camera image and a virtual camera image according to one embodiment of the present disclosure;

[0018] Figure 7 This is a schematic diagram of the polar correction principle according to one embodiment of the present disclosure;

[0019] Figure 8 This is a schematic diagram of the polar correction result according to one embodiment of the present disclosure;

[0020] Figure 9 This is a schematic diagram of the segmentation result according to one embodiment of the present disclosure;

[0021] Figure 10 This is a schematic diagram of a depth map according to one embodiment of the present disclosure;

[0022] Figure 11 This is a schematic diagram of reconstructing a 3D point cloud according to one embodiment of the present disclosure;

[0023] Figure 12 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure;

[0024] Figure 13 This is a schematic diagram of a non-volatile computer-readable storage medium according to one embodiment of the present disclosure. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] It should be noted that the steps shown can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0028] The camera calibration method according to one embodiment of the present disclosure may include the following steps:

[0029] Step S110: Acquire multiple captured images. The captured images are images of the calibration board taken by the camera from different distances and / or different angles. The calibration board contains calibration images, and the calibration images contain multiple discrete or continuous calibration points.

[0030] Step S130: Obtain the calibration points corresponding to each pixel of the camera in multiple captured images;

[0031] Step S150: Calculate the ray information of the corresponding pixel in the physical world based on the coordinates of the calibration point in the physical world, and obtain the set of ray information corresponding to each pixel of the camera to complete the calibration of the camera.

[0032] The camera calibration method in this exemplary embodiment has several advantages. First, the ray information of each pixel in the camera is independent, and they may or may not intersect at a single point, making the method more versatile and applicable not only to pinhole cameras but also to non-pinhole cameras. Second, this method calibrates each pixel of the camera independently, and the calibration result of each pixel is expressed by a ray information. Compared with the traditional calibration method that takes the entire camera as the calibration object and calculates the intrinsic and extrinsic parameters of camera distortion, this method subdivides the camera calibration to each pixel of the camera, and the calibration accuracy is much higher than that of the traditional calibration method. It can more accurately calibrate the distortion caused by various complex factors.

[0033] The camera calibration method in this exemplary embodiment will now be further described.

[0034] In step S110, multiple images are acquired. The images are images of the calibration board taken by the camera from different distances and / or different angles. The calibration board contains calibration images, and the calibration images contain multiple discrete or continuous calibration points.

[0035] In this exemplary embodiment, the camera can be a pinhole camera or a non-pinhole camera. Non-pinhole cameras include, but are not limited to: mobile phone cameras, light field cameras, fisheye lenses, wide-angle lenses, panoramic cameras, or other special-purpose non-pinhole cameras.

[0036] In an optional embodiment, during the shooting process in step S110, with the camera as the reference, multiple images are taken with the multiple planes on which the calibration plate is located parallel to each other. This facilitates the calculation of the ray information corresponding to each pixel in step S150.

[0037] In an optional implementation, during the shooting process in step S110, the optical axis of the camera may be perpendicular to the calibration plate or not.

[0038] In optional implementations, the calibration plate is fixed on a linear guide rail, and the camera can move along the linear guide rail; alternatively, the camera is fixed on the linear guide rail, and the calibration plate can move along the linear guide rail; or both the camera and the calibration plate can move along the linear guide rail. This makes the camera calibration process more convenient, allowing the operator to easily adjust the distance between the camera and the calibration plate to control the camera to capture multiple images of the calibration plate from different distances.

[0039] In an optional implementation, the calibration board can be the calibration board used in conventional camera calibration methods.

[0040] In an optional implementation, the calibration board is an electronic display screen device.

[0041] In optional implementations, the calibration board is an LCD panel, CRT display, or plasma display screen, and calibration images can be displayed on electronic display screen devices such as LCD panels, CRT displays, and plasma display screens. This can further improve the accuracy of camera calibration: firstly, electronic display screen devices such as LCD panels, CRT displays, and plasma display screens are flatter and have less physical distortion than traditional calibration boards; secondly, each pixel on electronic display screen devices such as LCD panels, CRT displays, and plasma display screens can serve as calibration points for camera calibration, and the number of pixels is far greater than the number of calibration points in traditional methods; thirdly, electronic display screen devices such as LCD panels, CRT displays, and plasma display screens have adjustable backlighting, resulting in higher imaging quality of the displayed calibration images compared to traditional calibration boards, and the pixels can also be considered as continuous calibration points.

[0042] In step S130, the calibration points corresponding to each pixel of the camera in the multiple captured images are obtained.

[0043] During the process of the camera taking pictures of the calibration board from different distances, any pixel in the camera corresponds to multiple calibration points on the calibration board (different calibration boards at different distances and / or different angles when the camera takes different pictures).

[0044] Different calibration plates at different distances and / or angles corresponding to the camera when capturing different images, or the same calibration plate at different distances and / or angles relative to the camera.

[0045] For example, such as Figure 1 As shown, the pixel (i, j) in plane I of the camera, with plane I as the coordinate reference, when multiple images (3 in this example) are captured, the calibration plate is located in planes π1, π2, and π3 respectively. The calibration point of pixel (i, j) in plane π1 is p1, the calibration point in plane π2 is p2, and the calibration point in plane π3 is p3.

[0046] For example, Figure 1 In the diagram, the distances between planes π1, π2, and π3 and plane I are 750mm, 800mm, and 850mm, respectively. Planes π1, π2, and π3 are all perpendicular to plane I, and the intersection of plane I and the Z-axis is the origin of the Z-axis.

[0047] The same principle applies to situations involving changing angles; only a few points on the ray need to be obtained. That is, the images can be obtained by taking pictures of the calibration plate from different distances and / or different angles. For example, when taking multiple pictures of the calibration plate from different angles, a rigid body transformation can be performed first, and then the same principle can be used to obtain a few points on the ray.

[0048] In an optional implementation, the captured image contains only the calibration image. This ensures that the captured image does not contain any background images other than the calibration image, allowing each pixel in the camera to correspond to a calibration point in the calibration image, thus guaranteeing the integrity and accuracy of camera calibration.

[0049] In step S150, the ray information of the corresponding pixel in the physical world is calculated based on the coordinates of the calibration point in the physical world, and the set of ray information corresponding to each pixel of the camera is obtained to complete the calibration of the camera.

[0050] For example, such as Figure 1As shown, the ray γ(i,j) of the corresponding pixel (i,j) in the physical world is calculated based on the coordinates of calibration points p1, p2, and p3 in the physical world. A set of ray information corresponding to each pixel of the camera is obtained to complete the camera calibration.

[0051] In an optional implementation, the ray information of each pixel can be calculated in parallel. As mentioned above, the ray information of each pixel is independent of each other. Thus, parallel computing can be used in the calculation of the ray information of each pixel to improve the data processing speed of camera calibration.

[0052] In an optional implementation, the ray information of the corresponding pixel in the physical world is calculated based on the coordinates of the calibration point in the physical world. Specifically, this includes calculating the ray information of the corresponding pixel in the physical world based on the coordinates of the calibration point in the physical world using Principal Component Analysis (PCA) or linear regression.

[0053] In an optional implementation, the ray information of the corresponding pixel in the physical world is calculated based on the coordinates of the calibration point in the physical world using Principal Component Analysis (PCA). Specifically, this includes: performing linear fitting based on the coordinates of the calibration point in the physical world using PCA or linear regression to calculate the ray information of the corresponding pixel in the physical world. Thus, on the one hand, the mathematical formulas in this camera calibration method are simpler than those of traditional methods; on the other hand, the PCA method seeks the global optimum, rather than the local optimum found in traditional nonlinear optimization.

[0054] In an optional implementation, one exemplary calculation process is as follows:

[0055] Suppose we are given in Given a set of points {p1, p2, p3, ..., pk}, our goal is to find a best-fit linear space π, which is called principal component analysis of the point set. Each point p... i Projecting onto a linear space yields a projection. Our goal is to minimize the approximation error.

[0056]

[0057] This is a least squares problem. For zero dimensions, the best-fit point c is obtained through the following optimization.

[0058]

[0059] By differentiating the energy, we obtain and central formula

[0060]

[0061] Then we will take each p i Displace to p i -c. For the best-fit line γ(t), it is expressed as

[0062]

[0063] Where d is the unit direction vector, and t is the directed distance from a point on the ray to the origin of the ray. It is the unit sphere in three-dimensional Euclidean space. Vector p i The projection of -c onto the best-fit line is <p i -c, d>d, the error vector is

[0064] e i :=(p i -c)- <p i -c, d>d,

[0065] Since the center c is fixed, the error vector e i Minimizing the length is equivalent to minimizing the projected component. <p i -c, d>d length is maximized. The signed length of the projected component is given by the following formula.

[0066] <p i -c, d>=d t (p i -c)=(p i -c) t d,

[0067] The least squares problem can then be formulated as maximizing the projected components:

[0068]

[0069] Where ∑ is the covariance matrix. By definition, for any vector

[0070]

[0071] Suppose we assume p i -c linearly generates the entire And v is not equal to 0, then v T Σv is positive. This indicates that the covariance matrix ∑ is positive definite. Therefore, it can be decomposed into...

[0072] ∑=O T ∑O,O T O = Id,

[0073] Where O is a rotation matrix, and is formed by the eigenvectors e of ∑.i composition,

[0074] O = (e1, e2, e3), Oe i =λ i , λ1≥λ2≥λ3,

[0075] Eigenvector formation orthogonal frame, <e i e j >=δ ij Therefore, the direction vector d can be expressed as d = x1e1 + x2e2 + x3e3.

[0076]

[0077] The equation holds if and only if d equals the first eigenvector e1 of ∑. This shows that the solution to the least squares problem in equation (2) is the first eigenvector of the covariance matrix. Therefore, the best-fit line is given by the following equation.

[0078] γ(t) = c + te1. (3)

[0079] In optional implementations, the calibration image is a structured light pattern such as a stripe pattern, Gray code pattern, or speckle pattern. This allows for the convenient and rapid acquisition of the physical coordinates of each calibration point by sequentially displaying structured light pattern patterns of different directions (e.g., horizontal and vertical), wavelengths, and phases on the calibration plate.

[0080] Phase-shifting algorithms can be used in both camera calibration and stereo matching processes. For example... Figure 2 As shown, a striped image can be projected onto the object to be photographed using a digital projector, or the striped pattern can be displayed on an LCD.

[0081] During the calibration process, the coordinates of each pixel on the LCD panel are encoded using the absolute phase of equations (5) and 5(a), and converted to intensity using equations (4) and (6). The camera captures the stripe image displayed on the LCD panel and calculates the pixel coordinates of the LCD panel.

[0082] For stereo matching, the 3D object is illuminated by a projected striped image. Each point on the 3D object corresponds to a pixel on the digital projector image. For each pixel in the camera image, its corresponding projector pixel can be calculated using a phase-shifting algorithm. Pixels on the left and right camera images are matched based on their corresponding projector pixels.

[0083] A digital projector was used as the light source, and then the projector displayed fringe patterns with different wavelengths and base phases. In our experiment, the horizontal fringe pattern of the projector was given by the following formula.

[0084]

[0085] Where a is the ambient component and b is the modulation coefficient.

[0086]

[0087] absolute phase

[0088]

[0089] λ is the wavelength. The absolute phase Φ(u, v) is determined solely by the horizontal coordinate u.

[0090] Similarly, the vertical stripe pattern is given by the following formula.

[0091]

[0092] in

[0093] Φ(u, v) = 2πv / λ, (5a)

[0094] Determined solely by the vertical coordinate v. The stripe pattern of a digital projector is as follows: Figure 3 As shown. During the calibration process, the LCD panel also displays a similar striped pattern.

[0095] Phase-shifting algorithms are widely used in 3D vision to capture depth from absolute phase. For single-wavelength phase-shifting algorithms, the phase is obtained using several fringe images with a certain phase shift. 3D coordinates are calculated based on the calibrated phase.

[0096] The three-step phase shift algorithm with a phase shift of 2π / 3 can be written as follows:

[0097] I1(x,y)=I′(x,y)+I″(x,y)cos[Φ(x,y)-2π / 3]

[0098] I2(x,y)=I′(x,y)+I″(x,y)cos[Φ(x,y)]

[0099] I3(x,y)=I′(x,y)+I″(x,y)cos[Φ(x,y)+2π / 3] (7)

[0100] Where I'(x,y) is the ambient component, I'(x,y) is the luminance modulation coefficient, and Φ(x,y) is the absolute phase. Figure 4 Three phase-shifted images are displayed.

[0101] Based on the three equations, we can obtain

[0102]

[0103] The relative phase or wrap phase is given by the following formula:

[0104]

[0105] φ(x, y) ranges from -π to π. Figure 5 Showing according to Figure 4 Calculated from the stripe image Figure 5 (a) Environment I′(x, y) Figure 5 (b) Modulation of I″(x,y) and Figure 5 (c) Wrapping Phase

[0106] The actual absolute phase is given by the following formula:

[0107]

[0108] Where k(x, y) is an integer. The process of recovering k(x, y) is called phase de-wrapping. In this work, we use a two-wavelength phase de-wrapping method.

[0109] Suppose we capture fringe images at two wavelengths λ1 and λ2, and calculate the wrapping phases Φ1(x,y) and Φ2(x,y). Then, according to equation (5),

[0110]

[0111] Where λ eq It is the equivalent wavelength.

[0112]

[0113] Assuming λ1 and λ2 are sufficiently close, therefore λ eq It is so large that the entire field of view (FOV) is covered by a single wavelength, therefore the wrapped phase φ1(x,y)-φ2(x,y) equals the absolute phase. This is achieved through the equivalent wavelength λ. eq The anti-wrapping phase can be used to recover the anti-wrapping phases of λ1 and λ2.

[0114] In an exemplary embodiment of this disclosure, an image correction method is also provided, which may include the following steps:

[0115] Step S210: Acquire a captured image using the camera;

[0116] Step S230: Obtain the set of ray information corresponding to each pixel of the camera;

[0117] Step S250: Generate a virtual correction image based on the captured images and the set.

[0118] This exemplary embodiment provides a novel image correction method that can synthesize a virtual camera image using an image captured by a physical camera and the ray information corresponding to each pixel of the physical camera, and can make the image distortion in the virtual camera image significantly less than the distortion in the physical camera image. Figure 6 This is a demonstration of experimental results from one of the optional implementation methods. Figure 6 (a) is a physical camera image. Figure 6 (b) is a virtual camera image where some circles are distorted into ellipses in the physical camera image, but are corrected to be circles in the virtual camera image.

[0119] In an optional implementation, a corrected image is generated based on the captured images and the set, specifically including:

[0120] The position of the correction pixel corresponding to each pixel is determined in the specified virtual imaging plane based on the ray information corresponding to each pixel.

[0121] The pixel value of the corrected pixel is determined based on the pixel value mapping of the pixel;

[0122] A virtual corrected image is generated based on the corrected pixels.

[0123] In an optional implementation, the virtual imaging plane may be parallel to the camera's imaging plane or may not be parallel to the camera's imaging plane.

[0124] Specifically, the position of the correction pixel corresponding to each pixel can be determined based on the interaction point between the ray corresponding to each pixel and the virtual imaging plane.

[0125] In an optional implementation, one exemplary calculation process is as follows:

[0126] First, define a virtual optical center. Since all camera rays may not necessarily intersect at a common point, the optical center can be defined as the center that minimizes the total squared distance to all camera rays. For each camera ray γ... ij := γ(i,j) can be represented as

[0127] γ ij (t): = a ij +n ij t,

[0128] Where a ij It is the base point, n ij It is a unit direction vector. From point to γ ij The square distance between lines is given by the Pythagorean theorem:

[0129]

[0130] Among them (pa) ij ) t n ij It is pa ij The projection onto the straight line. The sum of the squared distances to all lines is:

[0131]

[0132] To minimize the total squared distance, we differentiate it relative to p.

[0133]

[0134] Right now

[0135]

[0136] Then we get a linear equation.

[0137]

[0138] Where I is the identity matrix. Linear systems can be solved using the Eigen library. The virtual optical center is denoted as Ov.

[0139] Then, assume the virtual camera's imaging plane is π. v Each ray γ(i,j) of the camera intersects π at p(i,j). v Intersection. The (i, j)th pixel in the physical camera is mapped to p(i, j) in the virtual camera image plane. A piecewise linear mapping is used to map the image captured by the physical camera to the virtual camera image. This mapping can also be achieved directly using OpenGL's texture mapping methods. Figure 6 This shows the result of building a virtual camera. On the left is a real image taken by a physical camera, where some circles are distorted and appear elliptical. On the right is an image from the virtual camera, where all the circles have been corrected and appear more rounded.

[0140] In alternative implementations, in some scenarios, it may not be necessary to calculate the virtual optical center Ov. For example, only in scenarios where grayscale images are being corrected.

[0141] In an optional implementation, generating a virtual corrected image based on the captured images and the set includes:

[0142] The position of the mapped pixel corresponding to each pixel is determined in the specified virtual imaging plane based on the ray information corresponding to each pixel.

[0143] The pixel value of the mapped pixel is determined based on the pixel value mapping of the pixel.

[0144] The corrected pixels are obtained by sampling and interpolating multiple mapped pixels;

[0145] A virtual corrected image is generated based on the corrected pixels.

[0146] In an optional implementation, the set of cameras (the set of ray information corresponding to each pixel of the camera) is obtained by calibrating the cameras.

[0147] In an optional implementation, the ray information corresponding to each pixel of the camera is obtained by the camera calibration method in any of the above implementations.

[0148] In an exemplary embodiment of this disclosure, a stereo reconstruction method is also provided, which may include the following steps:

[0149] Step S310: Take a first image using the first camera;

[0150] Step S330: Obtain the first set of ray information corresponding to each pixel of the first camera;

[0151] Step S350: Take a second image using the second camera;

[0152] Step S370: Obtain the second set of ray information corresponding to each pixel of the second camera;

[0153] Step S390: Perform stereo reconstruction based on the first image, the second image, the first set, and the second set.

[0154] This exemplary embodiment provides a novel stereo reconstruction method that can perform stereo reconstruction using the ray information corresponding to each pixel of a first camera (left-eye camera) and a second camera (right-eye camera), and can make the reconstruction results more accurate.

[0155] It can be understood that the above-mentioned 3D reconstruction method can be practically applied to various fields such as autonomous driving, 3D modeling, medical treatment, and industrial inspection.

[0156] In an optional implementation, stereo reconstruction is performed based on the first image, the second image, the first set, and the second set, specifically including:

[0157] Determine the matching pixels of each pixel in the first image in the second image;

[0158] The location of the reconstructed pixel in the physical world is determined by comparing the ray information of each pixel in the first image with the ray information of the matching pixel in the second image.

[0159] In an optional implementation, the matching pixels of each pixel in the first image in the second image can be determined by using a stereo matching method in a traditional algorithm.

[0160] Traditional algorithms in stereo vision include correction, stereo matching, and triangulation (reconstruction). For example... Figure 7 As shown, the stereo matching algorithm matches each pixel in the left image with a unique pixel in the right image. The triangulation algorithm calculates the intersection points between camera rays passing through the matched pixel pairs to obtain depth information.

[0161] Epipolar Rectification

[0162] Finding matches in stereo vision is limited by epipolar geometry: such as Figure 8 As shown, if a pixel in one image is fixed, a matching pixel in another image can only be found on the so-called epipolar line. If the two optical centers differ by a horizontal displacement, the two images are coplanar, and the line connecting the optical centers is parallel, then the epipolar line for each pixel is horizontal, and the ordinate of the epipolar line is equal to the ordinate of the pixel. In general settings, the epipolar line is slanted. Image epipolar correction performs a projective transformation on the two images to keep their epipolar lines horizontal, thus simplifying the matching process.

[0163] Suppose we have two virtual cameras with optical centers O1 and O2, and image planes π1 and π2, respectively. We choose the midpoint of the optical centers as the origin of the corrected coordinate system.

[0164]

[0165] The new x-direction is given by the following formula.

[0166]

[0167] The corrected y-direction is given by the following formula.

[0168] e′2=e3×e′1

[0169] The corrected z-direction is equal to e′3 = e′1 × e′2, where × is the cross product of vectors. Then, we can transform the coordinate system to the corrected coordinate system {O′: e′1, e′2, e′3} using a rigid body transformation.

[0170]

[0171] Therefore, the coordinate transformation is given by the following equation.

[0172]

[0173] The translation vector is given by T = O′ - O, and the rotation matrix R is R = (e′1, e′2, e′3), which represents the coordinates of the axial quantities of the rectified frame.

[0174] In the corrected coordinate system, we choose the image plane π′ as z′=c′. Considering the virtual camera {π1, O1} on the left, each (i, j)th pixel on the image plane represents a ray γ(i, j) that intersects π′ at (u, v). This gives the projection transformation. (Using the optical center O1 as the projection center). Assuming that in the corrected coordinate system,

[0175] O′1=(x′1,y′1,0),

[0176] Then projection

[0177]

[0178] Similarly, we calculate the projection transformation.

[0179]

[0180] The projected image is the corrected image. The projection transformation from the virtual camera image to the corrected image can be performed using texture mapping in OpenGL. Figure 8 The results of the epipolar correction algorithm are shown.

[0181] Stereo-matching

[0182] The corrected environment component image is segmented into foreground and background using a simple intensity threshold. All pixels with an intensity less than the threshold ε are labeled as background, and all others are labeled as foreground. Figure 9 The segmentation results are displayed against a blue background. Stereo matching is performed only between foreground pixels.

[0183] In the corrected image on the left, for each foreground pixel (i, j), its corresponding epipolar line in the corrected image on the right is a horizontal line of equal height. We search along the epipolar lines pixel by pixel and find the matching pixel with the smallest difference; here we choose the one with the smallest absolute value of the absolute phase difference. In practice, we select a small neighborhood for each pixel and measure the total squared difference between the neighborhoods. The disparity map is defined as the difference between the horizontal indices of the matching pair. Figure 10 The disparity map calculated using this algorithm is shown.

[0184] Reconstruction (Triangulation)

[0185] In the corrected coordinate system, the optical centers of the left and right virtual cameras are (-1, 0, 0) and (+1, 0, 0) respectively, and both camera planes are z = c. The centers of the left and right camera images are (-1, 0, 0) and (+1, 0, 0). Assume the size of the camera image is wxh, and the distances between pixels are Δx and Δy. Then the coordinates of the pixel in the left image (i, j) and the coordinates of the pixel in the right image (i + f(i, j), j) are...

[0186] ((iw / 2)Δx+l, (jh / 2)Δy, c) ((i+f(i, j)-w / 2)Δx-l, (jh / 2)Δy, c),

[0187] Where f(i,j) is the disparity of pixel (i,j). Their intersection point between the two rays is given by the following equation.

[0188]

[0189] Therefore, 3D point clouds can be obtained accordingly. Figure 11 Displays a reconstructed 3D point cloud using stereo matching and triangulation.

[0190] In an optional implementation, the stereo reconstruction method further includes: determining the pixel value of the reconstructed pixel based on each pixel in the first image and / or the matching pixel in the second image.

[0191] In an optional implementation, the first set of the first camera (the set of ray information corresponding to each pixel of the first camera) is obtained by performing camera calibration on the first camera.

[0192] In an optional implementation, the second set of the second camera (the set of ray information corresponding to each pixel of the second camera) is obtained by performing camera calibration on the second camera.

[0193] In an optional implementation, the first set of the first camera (the set of ray information corresponding to each pixel of the first camera) is obtained by the camera calibration method in any of the above implementations.

[0194] In an optional implementation, the second set of the second camera (the set of ray information corresponding to each pixel of the second camera) is obtained by the camera calibration method in any of the above implementations.

[0195] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0196] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0197] The following reference Figure 12 To describe an electronic device 1200 according to such an embodiment of the present disclosure. Figure 12 The electronic device 1200 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0198] like Figure 12 As shown, the electronic device 1200 is manifested in the form of a general-purpose computing device. The components of the electronic device 1200 may include, but are not limited to: at least one processing unit 1210, at least one storage unit 1220, a bus 1230 connecting different system components (including storage unit 1220 and processing unit 1210), and a display unit 1240.

[0199] The storage unit stores program code that can be executed by the processing unit 1210, causing the processing unit 1210 to perform the steps described above according to various exemplary embodiments of this disclosure. The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) unit 12201 and / or a cache memory unit 12202, and may further include a read-only memory unit (ROM) 12203.

[0200] Storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0201] Bus 1230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0202] Electronic device 1200 can also communicate with one or more external devices 1270 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 1200, and / or with any device that enables electronic device 1200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1250. Furthermore, electronic device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 12120. As shown, network adapter 1260 communicates with other modules of electronic device 1200 via bus 1230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0203] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0204] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described above according to various exemplary embodiments of this disclosure.

[0205] refer to Figure 7 As shown, a program product 700 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0206] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0207] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0208] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0209] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0210] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0211] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0212] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0213] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A camera calibration method, characterized in that, The method includes: The system acquires multiple captured images, which are images of the calibration board taken by the camera from different distances and / or different angles. The calibration board contains calibration images, and the calibration images contain multiple discrete or continuous calibration points. The camera is a pinhole camera or a non-pinhole camera. The non-pinhole camera includes: mobile phone cameras, light field cameras, fisheye lenses, wide-angle lenses, panoramic cameras, or other special-purpose non-pinhole cameras. Obtain the calibration points corresponding to each pixel of the camera in the multiple captured images; The ray information of the corresponding pixel in the physical world is calculated based on the coordinates of the calibration point in the physical world, and the set of ray information corresponding to each pixel of the camera is obtained to complete the calibration of the camera. The step of calculating the ray information of the corresponding pixel in the physical world based on the coordinates of the calibration point in the physical world includes: calculating the ray information of the corresponding pixel in the physical world based on the coordinates of the calibration point in the physical world using principal component analysis or linear regression. The step of calculating the ray information of the corresponding pixel in the physical world based on the coordinates of the calibration point in the physical world using principal component analysis includes: performing linear fitting based on the coordinates of the calibration point in the physical world using principal component analysis or linear regression to calculate the ray information of the corresponding pixel in the physical world. The step of obtaining a set of ray information corresponding to each pixel of the camera to complete the calibration of the camera includes: independently calibrating each pixel of the camera, and expressing the calibration result of each pixel through a ray information.

2. The method according to claim 1, characterized in that, The calibration board is an electronic display screen device.

3. The method according to claim 2, characterized in that, The calibration board is an LCD panel, a CRT display screen, or a plasma display screen.

4. The method according to claim 1, characterized in that, The ray information of each pixel is calculated in parallel.

5. The method according to claim 1, characterized in that, The calibration image is a structured light pattern.

6. The method according to claim 5, characterized in that, The structured light pattern includes a stripe pattern, a Gray code pattern, or a speckle pattern.

7. The method according to claim 1, characterized in that, The captured images contain only the calibration images.

8. The method according to claim 1, characterized in that, The calibration plate is fixed on a linear guide rail, and the camera can move along the linear guide rail; or, the camera is fixed on a linear guide rail, and the calibration plate can move along the linear guide rail; or, both the camera and the calibration plate can move along the linear guide rail.

9. A method for stereoscopic reconstruction, characterized in that, The method includes: The first image is captured using the first camera; A first set of ray information corresponding to each pixel of the first camera is obtained by any of the camera calibration methods of claims 1 to 8; A second image is captured using a second camera; A second set of ray information corresponding to each pixel of the second camera is obtained by any of the camera calibration methods of claims 1 to 8; Stereoscopic reconstruction is performed based on the first image, the second image, the first set, and the second set.

10. The method according to claim 9, characterized in that, The stereo reconstruction based on the first image, the second image, the first set, and the second set includes: Determine the matching pixels of each pixel in the first image in the second image; The location of the reconstructed pixel in the physical world is determined based on the ray information of each pixel in the first image and the ray information of the matching pixel in the second image.

11. The method according to claim 10, characterized in that, The method includes: The pixel value of the reconstructed pixel is determined based on each pixel in the first image and / or the matching pixel in the second image.

12. The method according to claim 9, characterized in that, The ray information corresponding to each pixel of the first camera is obtained through the following method: Multiple captured images are obtained by the first camera from different distances and / or different angles to capture images of the calibration board. The calibration board contains calibration images, and the calibration images contain multiple discrete or continuous calibration points. Obtain the calibration points corresponding to each pixel of the first camera in the multiple captured images; The ray information of the corresponding pixel in the physical world is calculated based on the coordinates of the calibration point in the physical world.

13. An image correction method, characterized in that, The method includes: Capture images using a camera; The set of ray information corresponding to each pixel of the camera is obtained by any one of the camera calibration methods of claims 1 to 8; A virtual correction image is generated based on the captured images and the set.

14. The method according to claim 13, characterized in that, The step of generating a virtual corrected image based on the captured image and the set includes: The position of the correction pixel corresponding to each pixel is determined in the specified virtual imaging plane based on the ray information corresponding to each pixel. The pixel value of the corrected pixel is determined based on the pixel value mapping of the pixel; A virtual correction image is generated based on the correction pixels.

15. The method according to claim 13, characterized in that, The step of generating a virtual corrected image based on the captured image and the set includes: The position of the mapped pixel corresponding to each pixel is determined in the specified virtual imaging plane based on the ray information corresponding to each pixel. The pixel value of the mapped pixel is determined based on the pixel value mapping of the pixel; Corrected pixels are obtained by sampling and interpolating multiple mapped pixels; A virtual correction image is generated based on the correction pixels.

16. The method according to claim 13, characterized in that, The ray information corresponding to each pixel of the camera is obtained through the following method: Acquire multiple captured images, wherein the captured images are images of the calibration board obtained by the camera from different distances and / or different angles, the calibration board contains calibration images, and the calibration images contain multiple discrete or continuous calibration points; Obtain the calibration points corresponding to each pixel of the camera in the multiple captured images; The ray information of the corresponding pixel in the physical world is calculated based on the coordinates of the calibration point in the physical world.

17. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-16 by executing the executable instructions.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1-16.

Images