Epipolar correction methods, binocular cameras, and smart mobile devices

Abstract

This application relates to the field of stereo vision technology, disclosing an epipolar correction method, a binocular camera, and a smart mobile device. The method involves acquiring a first image and a second image including the same target. In both the first camera acquiring the first image and the second camera acquiring the second image, a coordinate axis in their respective camera coordinate systems is parallel to a baseline, which is the line connecting the optical centers of the first and second cameras. Then, a cylindrical projection model is established based on the optical centers of the first and second cameras. The first and second images are projected onto a preset arc surface of the cylindrical projection model and unfolded to obtain the corrected first and second cylindrical projection images. In this method, by projecting the first and second images onto the preset arc surface of the cylindrical projection model, the first and second cylindrical projection images are aligned in rows.

Description

Technical Field

[0001] This application relates to the field of stereo vision technology, and in particular to an epipolar correction method, a binocular camera, and a smart mobile device. Background Technology

[0002] Stereo matching involves finding corresponding points between pairs of stereo images. To improve search speed, the epipolar lines of the stereo image pairs must be on the same horizontal line, ensuring that the two cameras are calibrated into an ideal head-up binocular structure. In applications of stereo vision, epipolar line correction methods can be applied to stereo matching, camera pose processing in 3D television, and other areas.

[0003] Existing epipolar correction methods can be divided into two categories: those requiring camera calibration and those requiring weak calibration. Epipolar correction methods requiring camera calibration utilize camera calibration to obtain intrinsic and extrinsic parameters, including focal length, optical center coordinates, rotation matrix, and translation matrix, and then use physically meaningful methods to obtain the homography matrix. Weakly calibrated epipolar correction methods only use matching corresponding points from two images to solve for two suitable homography matrices and perform a projection transformation, ensuring that corresponding points have no vertical parallax.

[0004] Currently, for wide-angle cameras (such as fisheye and wide-angle cameras), it is impossible to project the image pixel edges during epipolar correction. In other words, there is no good solution for epipolar correction of wide-angle cameras. Summary of the Invention

[0005] The main technical problem addressed by the embodiments of this application is to provide an extreme correction method, a binocular camera, and a smart mobile device, which can simultaneously address epipolar correction and edge pixel projection for wide-angle cameras.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides an epipolar correction method, comprising:

[0007] Acquire a first image and a second image that include the same target, wherein in the camera coordinate system of the first camera used to acquire the first image and the second camera used to acquire the second image, one coordinate axis is parallel to the baseline, and the baseline is the line connecting the optical center of the first camera and the optical center of the second camera;

[0008] A cylindrical projection model is established based on the optical centers of the first and second cameras. The first and second images are projected onto the preset arc surface of the cylindrical projection model and unfolded to obtain the corrected first and second projected images, thereby achieving epipolar correction.

[0009] In some embodiments, the target image is either a first image or a second image. When the target image is the first image, the target cylindrical projection image is the first cylindrical projection image. When the target image is the second image, the target cylindrical projection image is the second cylindrical projection image.

[0010] The aforementioned cylindrical projection model is established based on the optical centers of the first and second cameras. The first and second images are then projected onto a preset arc surface of the cylindrical projection model and unfolded to obtain the corrected first and second projected images, including:

[0011] Based on the first pixel coordinates of the target pixel, obtain the second pixel coordinates of the target pixel mapped in the target image. Here, the target pixel is any pixel in the target cylindrical projection image, the coordinate system of the first pixel coordinates is the image coordinate system of the target cylindrical projection image, and the coordinate system of the second pixel coordinates is the image coordinate system of the target image.

[0012] Based on the target image and the second pixel coordinates, obtain the pixel value at the target pixel point.

[0013] In some embodiments, obtaining the second pixel coordinates of the target pixel mapped to the target image based on the first pixel coordinates of the target pixel includes:

[0014] Based on the first pixel coordinates of the target pixel, obtain the ray of the target pixel in the cylindrical projection model. The origin of the ray is the optical center of the target camera in the cylindrical projection model, and the direction of the ray is from the optical center of the target camera in the cylindrical projection model to the target pixel. The target camera is the camera used to capture the target image.

[0015] Obtain the angle between the ray and the optical axis, where the optical axis is the optical axis of the target camera used to acquire the target image;

[0016] The included angle is distorted to obtain the distorted included angle.

[0017] Based on the target camera's internal parameters, distortion angle, and ray coordinates, determine the second pixel coordinates of the target pixel in the target image.

[0018] In some embodiments, obtaining the ray of the target pixel in the cylindrical projection model based on the first pixel coordinates of the target pixel includes:

[0019] The following formula is used to calculate the rays;

[0020]

[0021] Where (X,Y,Z) are the coordinates of the ray, (w,h) are the coordinates of the first pixel, OW×OH is the resolution of the target cylindrical projection image, α is the viewing angle of the target camera, and β is the viewing angle of the effective projection of the target camera.

[0022] In some embodiments, determining the second pixel coordinates of the target pixel point mapped in the target image based on the target camera's internal parameters, distortion angle, and ray coordinates includes:

[0023] The coordinates of the second pixel are calculated using the following formula;

[0024]

[0025] dθ = distort(θ)

[0026] Where θ is the included angle, dθ is the distortion included angle, f is the focal length of the target camera, (c x c y ) represents the principal point coordinates of the target camera.

[0027] In some embodiments, obtaining the pixel value at the target pixel point based on the target image and the second pixel coordinates includes:

[0028] Take the pixel values ​​of the four pixels surrounding the second pixel coordinate position in the target image, and perform bilinear interpolation to obtain the pixel value at the target pixel.

[0029] To address the aforementioned technical problems, in a second aspect, embodiments of this application provide an epipolar correction device, comprising:

[0030] The acquisition module is used to acquire a first image and a second image that include the same target, wherein in the camera coordinate system of the first camera used to acquire the first image and the second camera used to acquire the second image, one coordinate axis is parallel to the baseline, and the baseline is the line connecting the optical center of the first camera and the optical center of the second camera;

[0031] The correction module is used to establish a cylindrical projection model based on the optical centers of the first camera and the second camera, and to project the first image and the second image onto the preset arc surface of the cylindrical projection model and unfold them to obtain the corrected first projection image and the second projection image, so as to achieve epipolar correction.

[0032] To address the aforementioned technical problems, in a third aspect, this application provides a binocular camera, including a first camera, a second camera, a processor, and a memory.

[0033] In this system, the camera coordinate axes of the first camera and the second camera are both parallel to the baseline, which is the line connecting the optical centers of the first camera and the second camera.

[0034] The processor is communicatively connected to the first camera and the second camera, respectively.

[0035] The memory is communicatively connected to the processor. The memory stores instructions that can be executed by the processor, and the processor executes the instructions to cause the processor to perform the method of the first aspect.

[0036] In some embodiments, the first camera and / or the second camera is a fisheye camera.

[0037] To address the aforementioned technical problems, in a fourth aspect, this application provides an intelligent mobile device, including a binocular camera as described in the third aspect.

[0038] The beneficial effects of this application's embodiments: Unlike the prior art, the epipolar correction method provided in this application first acquires a first image and a second image including the same target. In both the first camera used to acquire the first image and the second camera used to acquire the second image, one coordinate axis in their respective camera coordinate systems is parallel to a baseline. This baseline is the line connecting the optical centers of the first and second cameras, ensuring that the camera coordinate systems of the first and second cameras are aligned and that one coordinate axis in each camera coordinate system is parallel to the baseline. Then, a cylindrical projection model is established based on the optical centers of the first and second cameras. The first image and the second image are projected onto a preset arc surface of the cylindrical projection model and unfolded to obtain the corrected first and second cylindrical projection images. In the above method, by projecting the first image and the second image onto the preset arc surface of the cylindrical projection model, the first cylindrical projection image and the second cylindrical projection image are aligned in rows. That is, without adjusting the internal parameters of the camera (such as the focal length or principal point), the pixel y coordinates of the first camera and the second camera (binocular camera) can be aligned, which is simple and convenient. In addition, the edge pixels of the first image and the second image can be projected and corrected, which is suitable for epipolar correction with a large viewing angle. Attached Figure Description

[0039] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0040] Figure 1 This is a schematic diagram of a smart mobile device 100 in some embodiments of this application;

[0041] Figure 2 This is a schematic diagram of a first image and a second image before epipolar correction in some embodiments of this application;

[0042] Figure 3This is a schematic diagram of the first and second images after epipolar correction in some embodiments of this application;

[0043] Figure 4 This is a schematic diagram of the structure of the binocular camera 10 in some embodiments of this application;

[0044] Figure 5 This is a schematic diagram of the baseline between the first camera and the second camera in some embodiments of this application;

[0045] Figure 6 This is a flowchart illustrating the epipolar correction method in some embodiments of this application;

[0046] Figure 7 This is a schematic diagram of a cylindrical projection model in some embodiments of this application;

[0047] Figure 8 This is a schematic diagram illustrating the determination of target pixels in some embodiments of this application;

[0048] Figure 9 This is a schematic diagram of the polar correction device in some embodiments of this application. Detailed Implementation

[0049] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0052] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0053] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0054] Intelligent mobile devices are devices that move independently of direct human control, using methods such as remote control, autonomous or semi-autonomous operation. For example, they can navigate autonomously, avoid obstacles, and complete various tasks independently, relying on their own intelligent systems.

[0055] Depending on the application scenario, smart mobile devices can be drones, self-driving cars, robots, aerial photography equipment, etc. No restrictions are placed on smart mobile devices here; any electronic device that can move independently is sufficient.

[0056] It is understandable that, such as Figure 1 As shown, Figure 1 Taking a drone as an example, the intelligent mobile device 100 is used for illustrative purposes. The intelligent mobile device 100 is equipped with a binocular camera 10, which serves as its vision system. This vision system can perform basic functions similar to human eyes, providing reliable visual information for path planning. Target detection of obstacles and acquisition of their distance information are key components. The key to obstacle detection based on binocular stereo vision lies in: obstacle target extraction, i.e., identifying the position and size of the obstacle in the image; and stereo matching between image pairs of obstacle target areas to obtain the depth information of the obstacle target.

[0057] Specifically, stereo matching is the process of finding corresponding points (e.g., obstacles) in two different images of the same scene. This involves finding pairs of imaging points corresponding to the same spatial point (e.g., obstacle) in the first and second images (each captured by a binocular camera). Then, based on these correspondences, the spatial position of the object (e.g., obstacle) represented by the imaging points can be calculated (equivalent to 3D reconstruction). In other words, after obtaining the spatial position of the obstacle, distance and orientation information can be determined, enabling smart mobile devices to avoid obstacles.

[0058] A binocular camera consists of two cameras (a first camera and a second camera) at different positions. By moving or rotating the camera, it takes pictures of the same scene, obtaining a first image and a second image that includes obstacles.

[0059] like Figure 2 As shown in (a), O1 is the optical center of the first camera, and O2 is the optical center of the second camera. The line connecting O1 and O2 is called the baseline. Any target point P observed by the two cameras and two rays from the optical centers O1 and O2 define an epipolar plane. This epipolar plane intersects the imaging plane of the first image to form the first epipolar line I1, and intersects the imaging plane of the second image to form the second epipolar line I2. When the target point P moves in space, all epipolar lines I1 pass through the pole e1, and all epipolar lines I2 pass through the pole e2. The pole e1 is the intersection of the baseline and the imaging plane of the first image, and the pole e2 is the intersection of the baseline and the imaging plane of the second image.

[0060] It is understandable that the imaging point P1 of the target point P in the first image is on the first epipolar line I1, and the imaging point P2 of the target point P in the second image is on the second epipolar line I2.

[0061] Before epipolar correction, the first epipolar line I1 and the second epipolar line I2 are not parallel, causing the heights of imaging point P1 in the first image and imaging point P2 in the second image to be inconsistent. For example... Figure 2 As shown in (b), when matching point P in the first and second images, the heights of image point P1 and image point P2 are inconsistent, requiring a global search, which involves a large amount of computation.

[0062] After polar correction, such as Figure 3 As shown in (a), the first pole e1 and the second pole e2 are at infinity, virtually simulating that the optical axes of the first and second cameras are parallel, the first epipolar line I1 and the second epipolar line I2 are parallel, and the image points P1 and P2 are at the same height, thus achieving the goal of epipolar correction. Figure 3 As shown in (b), the pixels of the first and second images are aligned after correction. That is, the imaging point P1 of the target point P in the first image and the imaging point P2 of the target point P in the second image are in the same row of pixels. When performing subsequent stereo matching after correction, it is only necessary to search for matching points in the same row of pixels of the first and second images, which can greatly improve efficiency.

[0063] However, for wide-angle cameras (such as fisheye and wide-angle cameras), it is impossible to project the image pixel edges when performing epipolar correction. In other words, there is no good solution for epipolar correction for wide-angle cameras.

[0064] In view of this, this application provides an epipolar correction method applied to the binocular camera, enabling the binocular camera to automatically perform epipolar correction.

[0065] In some embodiments, please refer to Figure 4 The binocular camera 10 includes a first camera 11, a second camera 12, a processor 13, and a memory 14.

[0066] Among them, such as Figure 5 As shown, the camera coordinate axes of the first camera 11 and the second camera 12 are both parallel to the baseline, which is the line connecting the optical center of the first camera 11 and the optical center of the second camera 12.

[0067] In some embodiments, the first or second camera may be a fisheye camera, which has a wide field of view, creating conditions for close-up shooting of a wide range of scenes.

[0068] The processor is communicatively connected to the first camera, the second camera, and the memory. Figure 4 The example is illustrated using a bus connection.

[0069] The processor 13 is used to provide computing and control capabilities to control the binocular camera 10 to perform corresponding tasks, such as controlling the binocular camera 10 to perform any of the epipolar correction methods provided in the following embodiments of the invention.

[0070] It is understood that processor 13 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0071] Memory 14, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the epipolar correction method in the embodiments of this application. Processor 13 can implement the epipolar correction method in any of the following method embodiments by running the non-transitory software programs, instructions, and modules stored in memory 14. Specifically, memory 14 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 14 may also include memories remotely located relative to the processor, and these remote memories can be communicatively connected to the processor. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0072] The epipolar correction method in the embodiments of this application is described in detail below. Please refer to [link / reference]. Figure 6 The method includes, but is not limited to, the following steps:

[0073] S10: Acquire a first image and a second image that include the same target.

[0074] It is understandable that the same target refers to the same scenario, such as the one described above. Figure 2 Obstacles, etc., as shown in (b).

[0075] The first image was captured by the first camera, and the second image was captured by the second camera. (As above) Figure 5 As shown, the baseline is the line connecting the optical centers of the first camera and the second camera. In the camera coordinate system of the first camera, one coordinate axis (e.g., the x-axis) is parallel to the baseline, and in the camera coordinate system of the second camera, one coordinate axis (e.g., the x-axis) is parallel to the baseline.

[0076] By setting the camera coordinate system for the first and second cameras as described above, the external parameters of the first and second cameras are made consistent, which facilitates the calculation of subsequent limit correction.

[0077] S20: Establish a cylindrical projection model based on the optical centers of the first and second cameras, project the first and second images onto the preset arc surface of the cylindrical projection model and unfold them to obtain the corrected first and second projected images, thereby achieving epipolar correction.

[0078] like Figure 7 As shown, in the cylindrical projection model, point A represents the optical center of the first camera, point B represents the optical center of the second camera, AD represents the optical axis of the first camera, AC represents the optical axis of the second camera, and quadrilateral GHEF represents the target image to be projected (either the first or second image). After projection, it becomes the target cylindrical projection image JDILCK. It is understood that the default curved surface is the curved surface JILK.

[0079] The target image is projected onto the preset curved surface JILK of the cylindrical projection model and unfolded to obtain the target cylindrical projection image. Here, the target image can be either the first image or the second image. It can be understood that when the target image is the first image, the corresponding target cylindrical projection image is the first cylindrical projection image. When the target image is the second image, the corresponding target cylindrical projection image is the second cylindrical projection image.

[0080] Specifically, the first image is projected onto a pre-set curved surface (JILK) on the cylindrical surface, and then the projected image on the cylindrical surface is unfolded to obtain the first cylindrical surface projected image. The second image is projected onto a pre-set curved surface (JILK) on the cylindrical surface, and then the pattern image on the cylindrical surface is unfolded to obtain the second cylindrical surface projected image.

[0081] Taking the first image resolution as x*y as an example, ∠GAH is the viewing angle of the first camera on the y-axis, that is, the y-axis is the vertical axis y in the resolution x*y. Assuming the target image's y-axis resolution is 640, then ∠GAH needs to be divided into 640 equal parts. When performing cylindrical projection, the x-axis of the target image (which can be either the first or second image) is parallel to the central axis of the cylinder, and the y-axis of the resolution lies within the arc defined by ∠GAH. Therefore, one equally divided angle of ∠GAH corresponds to one pixel in the y-axis of the resolution.

[0082] For a 3D point M, MA intersects the cylinder at point M1, MB intersects the cylinder at point M2, the angle between plane ABM1 and plane ABCD is λ, and the angle between plane ABM2 and plane ABCD is λ. Where, λ and Since they are equal, the first cylindrical projection image and the second cylindrical projection image are aligned on the y-axis.

[0083] The above method enables row alignment of the corrected first and second cylindrical projection images. This means that the pixel y-coordinates of the first and second cameras (binocular cameras) can be aligned without needing to adjust internal camera parameters (e.g., focal length or principal point consistency), which is simple and convenient. Furthermore, in the cylindrical projection model, the pixel x-coordinate of the first or second image is projected onto the vertical direction of the cylinder, and the pixel y-coordinate is projected onto the circumference. Therefore, projection correction can be performed on either the first or second image when the field of view is large, including edge pixels, making it suitable for epipolar correction of wide-angle cameras.

[0084] In some embodiments, step S20 specifically includes:

[0085] S21: Based on the first pixel coordinates of the target pixel, obtain the second pixel coordinates of the target pixel mapped in the target image.

[0086] Here, the target pixel is any pixel in the target cylindrical projection image. The projection process of the target pixel is illustrated here. It can be understood that each pixel in the target cylindrical projection image is projected in the following manner.

[0087] For example, the target pixel is a pixel on the target cylindrical projection image, and its first pixel coordinates on the target cylindrical projection image are (w, h). Since the target image is projected onto the cylinder, and the projected image on the cylinder is unfolded to obtain the target cylindrical projection image, the second pixel coordinates (u, v) of the target pixel in the target image can be determined based on the above projection relationship.

[0088] In some implementations, step S21 specifically includes:

[0089] S211: Obtain the ray of the target pixel in the cylindrical projection model based on the first pixel coordinates of the target pixel.

[0090] In some embodiments, the rays can be calculated using the following formula;

[0091]

[0092] Where (X,Y,Z) are the coordinates of the ray, (w,h) are the coordinates of the first pixel, OW×OH is the resolution of the target cylindrical projection image, α is the viewing angle of the target camera, β is the effective projection viewing angle of the target camera, and β≤α. The origin of the ray's coordinates is the optical center of the target camera in the cylindrical projection model, and the direction of the ray is from the optical center of the target camera in the cylindrical projection model to the target pixel. The target camera is the camera used to capture the target image.

[0093] Understandably, please refer to it again. Figure 7 In the cylindrical projection model, point A represents the optical center of the first camera, point B represents the optical center of the second camera, AD represents the optical axis of the first camera, AC represents the optical axis of the second camera, and quadrilateral GHEF represents the target image to be projected, which becomes the target cylindrical projection image JDILCK after projection.

[0094] The target pixel (w, h) is a pixel on the target cylindrical projection image JDILCK. Taking the target cylindrical projection image JDILCK as the first cylindrical projection image, the target image as the first image, and the target camera as the first camera A as an example, the target pixel is point M in the cylindrical projection model, then the ray is AM, and (X, Y, Z) calculated by the above formula are the coordinates of ray AM.

[0095] S212: Obtain the angle between the ray and the optical axis, where the optical axis is the optical axis of the target camera used to acquire the target image.

[0096] Understandably, this included angle can be calculated using the following formula.

[0097] (X,Y,Z) represents a ray.

[0098] Please refer to it again. Figure 7 θ is the angle between ray AM and optical axis AD.

[0099] S213: Distort the above included angle, that is, adjust the angle size to obtain the distorted included angle. That is, the distorted included angle dθ = distort(θ).

[0100] It is understandable that, since images taken by real cameras are distorted, while virtual cameras corresponding to cylindrical projection images are not distorted, distortion is added to the aforementioned angles in order to conform to the characteristics of real cameras.

[0101] S214: Determine the second pixel coordinates of the target pixel in the target image based on the target camera's internal parameters, distortion angle, and ray coordinates.

[0102] Here, the internal parameters can include focal length and camera principal point. For example, the second pixel coordinates (u,v) corresponding to the target pixel in the target image can be calculated using the following formula.

[0103]

[0104] Where θ is the included angle, dθ is the distortion included angle, f is the focal length of the target camera, (c x c y ) represents the principal point coordinates of the target camera.

[0105] In this embodiment, by sequentially calculating the ray of the target pixel in the cylindrical projection model, the angle between the ray and the optical axis, and then using the angle, the principal point of the camera, the ray, and the focal length, the second pixel coordinates corresponding to the target pixel in the target image are calculated. That is, a correspondence table between the pixel coordinates of the original target image and the pixel coordinates of the target cylindrical projection image is established. Furthermore, by looking up the table, the pixel coordinates corresponding to the previous pixel in the target cylindrical projection image can be obtained on the target image, and then the pixel value of the previous pixel in the cylindrical projection image can be obtained, which enables a faster acquisition of the pixel-aligned image.

[0106] S22: Obtain the pixel value at the target pixel point based on the target image and the second pixel coordinates.

[0107] It is understandable that, such as Figure 8 As shown, the second pixel coordinates are the pixel coordinates (u,v) of the target pixel (w,h) in the target image. Given the pixel values ​​at each pixel coordinate in the target image, we can find multiple pixel values ​​near the second pixel coordinates (u,v) in the target image. Then, based on these multiple pixel values ​​near the second pixel coordinates (u,v) in the target image, we can determine the pixel value (w,h) of the target pixel in the cylindrical projection image.

[0108] In some embodiments, step S22 specifically includes:

[0109] S221: Take the pixel values ​​of the four pixels surrounding the second pixel coordinate position in the target image, and perform bilinear interpolation to obtain the pixel value at the target pixel.

[0110] Please refer to it again. Figure 8 The image is illustrated using the bottom left corner as the origin of the image coordinate system. The second pixel's coordinates are defined as point Q. It's important to understand that the coordinates of point Q obtained through mapping may not be integers (i.e., not entire rows or columns). Therefore, we can use four pixels Q''s nearest neighbor. 12 Q 13 Q 22 Q 23 For each pixel at point Q, bilinear interpolation is performed to calculate the pixel value at point Q.

[0111] Specifically, by Q 12 and Q 22 The pixel value at point R2 is obtained by performing single linear interpolation in the x-direction, and then using Q... 13 and Q 23 The pixel value at point R3 is obtained by performing single linear interpolation in the x-direction from the pixel values ​​at point R2 and R3. Then, the pixel value at point Q is obtained by performing single linear interpolation in the y-direction from the pixel values ​​at points R2 and R3.

[0112] In this embodiment, considering the non-integer nature of the second pixel coordinates obtained by mapping, bilinear interpolation is used to calculate the pixel value at the target pixel point, which is more accurate and ensures that the corrected target cylindrical projection image is not distorted.

[0113] In summary, the epipolar correction method provided in this application first acquires a first image and a second image containing the same target. In both the first camera used to acquire the first image and the second camera used to acquire the second image, one coordinate axis is parallel to a baseline. This baseline is the line connecting the optical centers of the first and second cameras, ensuring that the camera coordinate systems of the first and second cameras are aligned and that one coordinate axis in each camera coordinate system is parallel to the baseline. Then, a cylindrical projection method is used to perform projection correction on the first image to obtain a corrected first cylindrical projection image, and the second image is also subjected to projection correction to obtain a corrected second cylindrical projection image. In this method, cylindrical projection enables row alignment of the first and second cylindrical projection images. This means that the pixel y-coordinates of the first and second cameras (binocular cameras) can be aligned without needing to adjust internal camera parameters (e.g., focal length or principal point consistency), which is simple and convenient. Furthermore, projection correction can be performed on the edge pixels of the first and second images, enabling epipolar correction for a wide-angle camera.

[0114] Another embodiment of this application also provides a polar correction device; please refer to [link to relevant documentation]. Figure 9 The polar line correction device 300 includes an acquisition module 301 and a correction module 302.

[0115] The acquisition module 301 is used to acquire a first image and a second image that include the same target, wherein a coordinate axis in the camera coordinate system of the first camera used to acquire the first image and the second camera used to acquire the second image is parallel to a baseline, which is the line connecting the optical center of the first camera and the optical center of the second camera.

[0116] The correction module 302 is used to establish a cylindrical projection model based on the optical center of the first camera and the optical center of the second camera, and to project the first image and the second image onto the preset arc surface of the cylindrical projection model and unfold them to obtain the corrected first projection image and the second projection image, so as to realize epipolar correction.

[0117] The aforementioned apparatus first acquires a first image and a second image containing the same target. In both the first camera acquiring the first image and the second camera acquiring the second image, one coordinate axis in their camera coordinate systems is parallel to a baseline. This baseline is the line connecting the optical centers of the first and second cameras, ensuring that the camera coordinate systems of the first and second cameras are aligned and that one coordinate axis in each camera coordinate system is parallel to the baseline. Then, using cylindrical projection, the first image is corrected to obtain a corrected first cylindrical projection image, and the second image is corrected to obtain a corrected second cylindrical projection image. In this method, cylindrical projection achieves row alignment between the first and second cylindrical projection images. This means that the pixel y-coordinates of the first and second cameras (binocular cameras) can be aligned without needing to adjust internal camera parameters (e.g., focal length or principal point consistency), which is simple and convenient. Furthermore, it can also perform projection correction on the edge pixels of the first and second images, enabling epipolar correction for a wide-angle camera.

[0118] It should be noted that, Figure 9 For any content not mentioned in the corresponding embodiments, please refer to the description of the above method embodiments, which will not be repeated here.

[0119] Another embodiment of this application provides a non-volatile computer-readable storage medium storing computer-executable instructions that, when executed by at least one processor, cause the at least one processor to perform the polar correction method as described in the foregoing embodiments.

[0120] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method of polar line correction, characterized by, The method comprises: acquiring a first image and a second image of the same target, wherein a camera coordinate system of a first camera used for acquiring the first image has a coordinate axis parallel to a baseline, a camera coordinate system of a second camera used for acquiring the second image has a coordinate axis parallel to the baseline, and the baseline is a line connecting a principal point of the first camera and a principal point of the second camera; wherein the first camera and / or the second camera is a fisheye camera; establishing a cylindrical projection model according to the principal point of the first camera and the principal point of the second camera, projecting the first image and the second image into a preset curved surface of the cylindrical projection model respectively and unfolding to obtain a corrected first projection image and a corrected second projection image, so as to realize polar correction; the establishing of the cylindrical projection model according to the principal point of the first camera and the principal point of the second camera, the projecting of the first image and the second image into the preset curved surface of the cylindrical projection model respectively and the unfolding to obtain the corrected first projection image and the corrected second projection image comprise: acquiring a second pixel coordinate corresponding to a target pixel point in a target image according to a first pixel coordinate of the target pixel point, wherein the target pixel point is any one pixel point in a target cylindrical projection image, a coordinate system of the first pixel coordinate is an image coordinate system of the target cylindrical projection image, and a coordinate system of the second pixel coordinate is an image coordinate system of the target image; acquiring a pixel value at the target pixel point according to the target image and the second pixel coordinate.

2. The method of claim 1, wherein, The target image is any one of the first image and the second image, the target cylindrical projection image is a first cylindrical projection image when the target image is the first image, and the target cylindrical projection image is a second cylindrical projection image when the target image is the second image.

3. The method of claim 1, wherein, the acquiring of the second pixel coordinate corresponding to the target pixel point in the target image according to the first pixel coordinate of the target pixel point comprises: acquiring a ray of the target pixel point in the cylindrical projection model according to the first pixel coordinate of the target pixel point, a coordinate origin of the ray is a principal point of a target camera in the cylindrical projection model, and a direction of the ray is a direction from the principal point of the target camera to the target pixel point in the cylindrical projection model, wherein the target camera is a camera used for shooting the target image; acquiring an included angle between the ray and an optical axis, wherein the optical axis is an optical axis of the target camera used for acquiring the target image; performing distortion processing on the included angle to obtain a distorted included angle; determining the second pixel coordinate corresponding to the target pixel point in the target image according to an internal parameter of the target camera, the distorted included angle and a coordinate of the ray.

4. The method of claim 3, wherein, the acquiring of the ray of the target pixel point in the cylindrical projection model according to the first pixel coordinate of the target pixel point comprises: the ray is calculated by using the following formula; wherein (X, Y, Z) is the coordinate of the ray, (w, h) is the first pixel coordinate, is the resolution of the target cylindrical projection image, is the angle of view of the target camera, is the angle of view of the effective projection of the target camera.

5. The method of claim 4, wherein, the determining of the second pixel coordinate corresponding to the target pixel point in the target image according to the internal parameter of the target camera, the distorted included angle and the coordinate of the ray comprises: The second pixel coordinate is calculated by using the following formula: where θ is the angle, dθ is the distortion angle, f is the focal length of the target camera, is the principal point coordinate of the target camera.

6. The method of claim 1, wherein, The pixel value at the target pixel point is obtained according to the target image and the second pixel coordinate, and the method comprises: The pixel values of the four pixel points around the second pixel coordinate position in the target image are taken, and bilinear interpolation is performed to obtain the pixel value at the target pixel point.

7. A radial line correction device, characterized by Comprise: The first image and the second image of the same target are obtained by using a first camera and a second camera, wherein the camera coordinate system of the first camera has a coordinate axis parallel to a baseline, the camera coordinate system of the second camera has a coordinate axis parallel to the baseline, and the baseline is a line connecting the optical center of the first camera and the optical center of the second camera; The cylindrical projection model is established according to the optical center of the first camera and the optical center of the second camera, the first image and the second image are projected into the preset curved surface of the cylindrical projection model and unfolded to obtain the corrected first projection image and the second projection image, so as to realize the polar line correction; wherein the first camera and / or the second camera is a fisheye camera; The cylindrical projection model is established according to the optical center of the first camera and the optical center of the second camera, the first image and the second image are projected into the preset curved surface of the cylindrical projection model and unfolded to obtain the corrected first projection image and the second projection image, so as to realize the polar line correction; wherein the first camera and / or the second camera is a fisheye camera; According to the first pixel coordinate of the target pixel point, the corresponding second pixel coordinate of the target pixel point in the target image is obtained, wherein the target pixel point is any one pixel point in the target cylindrical projection image, the coordinate system of the first pixel coordinate is the image coordinate system of the target cylindrical projection image, and the coordinate system of the second pixel coordinate is the image coordinate system of the target image; The pixel value at the target pixel point is obtained according to the target image and the second pixel coordinate.

8. A binocular camera, characterized by Comprise a first camera, a second camera, a processor and a memory; the first camera and / or the second camera is a fisheye camera; The camera coordinate system of the first camera has a coordinate axis parallel to a baseline, the camera coordinate system of the second camera has a coordinate axis parallel to the baseline, and the baseline is a line connecting the optical center of the first camera and the optical center of the second camera; The processor is in communication connection with the first camera and the second camera respectively; The memory is in communication connection with the processor, the memory stores instructions executable by the processor, and the instructions are executed by the processor to make the processor execute the method of any one of claims 1-6.

9. A smart mobile device, characterized by Comprise the binocular camera of claim 8.

Images