Dual camera secondary calibration method and apparatus

Abstract

The application discloses a binocular camera secondary calibration method and device, the method comprises the following steps: taking two target targets by a camera to be calibrated; extracting left target corner points of the two target targets in a left image respectively, and extracting right target corner points of the two target targets in a right image respectively; extracting a left vanishing point in the left image according to the left target corner points of the two target targets, and extracting a right vanishing point in the right image according to the right target corner points of the two target targets; calculating a pitch angle and a yaw angle of a rotation matrix between the left eye camera and the right eye camera according to the left vanishing point and the right vanishing point; calculating left three-dimensional coordinates and right three-dimensional coordinates of the target corner points based on a monocular principle; and calculating a translation matrix between the left eye camera and the right eye camera according to the left three-dimensional coordinates and the right three-dimensional coordinates. In the automatic driving or assisted driving scene, the calculation accuracy of the rotation matrix and the translation matrix is improved, so that the accuracy of the binocular camera secondary calibration is improved.

Description

Technical Field

[0001] This invention relates to the field of camera calibration technology, and specifically to a method and apparatus for secondary calibration of a binocular camera. Background Technology

[0002] With the increasing prevalence of intelligent technology, autonomous driving has become a crucial development project in cutting-edge science and technology, involving machine vision applications. Camera parameter calibration is a critical step in this process. Binocular camera calibration primarily determines the camera's intrinsic and extrinsic parameters. The accuracy of the calibration results and the stability of the algorithm directly affect subsequent calculations of the object's 3D position, depth, and distance, thus impacting the accuracy of 3D reconstruction algorithms and other results. Therefore, accurate binocular camera calibration is a prerequisite for ensuring the accuracy of subsequent algorithms, and improving calibration accuracy is of great significance in advanced driver assistance systems (ADAS) for intelligent vehicles.

[0003] In practical applications, such as autonomous driving assistance systems, the ranging error must be less than 5% within 100 meters. Therefore, secondary calibration of the binocular cameras is generally chosen to calibrate more precise rotation and translation matrices to ensure that the ranging accuracy meets the requirements of the scenario. In addition, in autonomous driving scenarios, multi-view cameras are usually mounted behind the vehicle windshield, and the tilt angle of the windshield itself can damage the already calibrated cameras, thus requiring secondary calibration even more.

[0004] The basic scheme for binocular camera calibration is to use a chessboard target with different spatial poses to simultaneously calibrate the intrinsic parameters and distortion coefficients of a single camera, as well as the rotation and translation matrices between multiple cameras. Currently, the calculation principle of commonly used binocular camera calibration methods involves fitting multiple corner points and multiple poses. This results in the need for relatively large targets and the requirement to rotate the target pose while maintaining a fixed position, making them suitable only for calibration under indoor or laboratory conditions. Furthermore, because the longitudinal position of the target relies on manual judgment and fixation, the calculation accuracy of the rotation and translation matrices between multiple cameras is poor, leading to a significant decrease in long-distance ranging accuracy. Summary of the Invention

[0005] Therefore, embodiments of the present invention provide a method and apparatus for secondary calibration of a binocular camera, aiming to improve the calculation accuracy of rotation and translation matrices in autonomous driving or assisted driving scenarios, thereby improving the accuracy of secondary calibration of the binocular camera.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A method for secondary calibration of a binocular camera, the method comprising:

[0008] Two target images are captured by the left eye camera of the camera to be calibrated to obtain a left image of the two target images; two target images are captured by the right eye camera of the camera to be calibrated to obtain a right image of the two target images.

[0009] In the left image, extract the left corner points of the two target objects respectively; in the right image, extract the right corner points of the two target objects respectively.

[0010] Extract the left vanishing point in the left image based on the left corner of the two target objects, and extract the right vanishing point in the right image based on the right corner of the two target objects;

[0011] Based on the left vanishing point and the right vanishing point, calculate the pitch angle and yaw angle of the rotation matrix between the left eye camera and the right eye camera;

[0012] Calculate the left and right three-dimensional coordinates of the target corner points based on the monocular principle;

[0013] Calculate the translation matrix between the left and right cameras based on the left and right three-dimensional coordinates.

[0014] In some embodiments, the pitch and yaw angles of the rotation matrix between the left and right cameras are calculated, followed by:

[0015] Connect all the corresponding target corner points of the two targets in the left figure to obtain the equation of the left straight line; connect all the corresponding target corner points of the two targets in the right figure to obtain the equation of the right straight line.

[0016] The roll angle of the rotation matrix is ​​calculated based on the equations of the left and right lines.

[0017] In some embodiments, connecting all corresponding target corner points of the two target targets in the left figure with a line, the process further includes:

[0018] Rotate the left image using a preset rotation matrix.

[0019] In some embodiments, the expression for the equation of the left straight line is:

[0020] in, Represents the vertical coordinates in the left image. Represents the horizontal coordinates in the left image. This represents the slope of the line equation in the left figure. This represents the intercept of the line equation in the left figure. In some embodiments, the expression for the right line equation is:

[0021] in, Represents the vertical coordinates in the right-hand image. This represents the horizontal coordinate in the right image. This represents the slope of the line equation in the right-hand graph. This represents the intercept of the equation of the line in the right figure.

[0022] In some embodiments, the left three-dimensional coordinates of the target corner point are calculated using a first preset formula, and the right three-dimensional coordinates of the target corner point are calculated using a second preset formula.

[0023] The first preset formula is:

[0024] In the formula, L1 is the three-dimensional spatial distance between the top corners of the two target points in the left image, and P1 is the image distance between the top corners of the two target points in the left image. The three-dimensional distance between the target corner points in the left image is... The left three-dimensional coordinates of the target corner point, and F1 represents the focal length of the left camera;

[0025] The second preset formula is:

[0026] In the formula, L2 is the three-dimensional spatial distance between the top corners of the two target points in the right figure, and P2 is the image distance between the top corners of the two target points in the right figure. The three-dimensional distance is the target corner point in the right figure. F1 represents the right three-dimensional coordinates of the target corner point, and F2 represents the focal length of the right camera.

[0027] The present invention also provides a secondary calibration device for a binocular camera, the device comprising:

[0028] The image acquisition unit is used to capture images of two target objects using the left eye camera of the camera to be calibrated, so as to obtain a left image of the two target objects; and to capture images of the two target objects using the right eye camera of the camera to be calibrated, so as to obtain a right image of the two target objects.

[0029] The corner point extraction unit is used to extract the left corner points of the two target targets in the left image and the right corner points of the two target targets in the right image.

[0030] The vanishing point extraction unit is used to extract the left vanishing point in the left image based on the left target corner point of the two target objects, and to extract the right vanishing point in the right image based on the right target corner point of the two target objects.

[0031] The rotation matrix calibration unit is used to calculate the pitch angle and yaw angle of the rotation matrix between the left eye camera and the right eye camera based on the left vanishing point and the right vanishing point;

[0032] The three-dimensional coordinate calculation unit is used to calculate the left and right three-dimensional coordinates of the target corner points based on the monocular principle.

[0033] The translation matrix calibration unit is used to calculate the translation matrix between the left and right cameras based on the left three-dimensional coordinates and the right three-dimensional coordinates.

[0034] The present invention also provides a smart terminal, the smart terminal comprising: a data acquisition device, a processor, and a memory;

[0035] The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions to perform the method described above.

[0036] The present invention also provides a computer-readable storage medium comprising one or more program instructions for performing the method described above.

[0037] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0038] The binocular camera secondary calibration method and apparatus provided by this invention involves capturing images of two target objects using the left eye camera of the camera to be calibrated, resulting in a left image of the two target objects; capturing images of the two target objects using the right eye camera of the camera to be calibrated, resulting in a right image of the two target objects; extracting the left corner points of the two target objects from the left image and the right corner points of the two target objects from the right image; extracting the left vanishing point from the left image based on the left corner points and the right vanishing point from the right image based on the right corner points; calculating the pitch and yaw angles of the rotation matrix between the left and right eye cameras based on the left and right vanishing points; calculating the left and right three-dimensional coordinates of the target corner points based on the monocular principle; and calculating the translation matrix between the left and right eye cameras based on the left and right three-dimensional coordinates. This invention utilizes the vanishing point principle of perspective, the image ranging principle, and the multi-view 3D coordinate difference to perform secondary calibration of the epipolar constraint model of a binocular camera. In autonomous driving or assisted driving scenarios, it improves the calculation accuracy of the rotation matrix and translation matrix, thereby improving the accuracy of the secondary calibration of the binocular camera. Attached Figure Description

[0039] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0040] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0041] Figure 1 This is one of the flowcharts for the binocular camera secondary calibration method provided by the present invention;

[0042] Figure 2 This is a schematic diagram of the optical center point in a specific application scenario of the present invention;

[0043] Figure 3 This is a schematic diagram of a general model of a binocular camera in a specific application scenario of the present invention;

[0044] Figure 4 This is a schematic diagram of the parallel phase plane model of a stereo camera in a specific application scenario of the present invention.

[0045] Figure 5 This is a schematic diagram of a concentric and coplanar model of a binocular camera in a specific application scenario of the present invention;

[0046] Figure 6 This is the second flowchart of the binocular camera secondary calibration method provided by the present invention;

[0047] Figure 7 This is a scene diagram of target placement position 1 in a specific application scenario of the present invention;

[0048] Figure 8 This is a scene diagram showing the target placement position 2 in a specific application scenario of the present invention;

[0049] Figure 9 This is a schematic diagram illustrating the principle of determining the intersection of parallel lines at the vanishing point of an image at a specific application scenario of the present invention.

[0050] Figure 10 This is a structural block diagram of the binocular camera secondary calibration device provided by the present invention. Implementation

[0051] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] First, the terminology involved in the embodiments of the present invention will be explained.

[0053] Parallax: The positional deviation of pixels in the same scene when imaged by two cameras, left and right. For example, point X in the scene has an x-coordinate in the left camera and an (x+d) coordinate in the right camera. Then d is the value of the x-coordinate point in the parallax map, and the parallax of each point on the parallax map is denoted as disp.

[0054] Camera calibration: In machine vision applications, to determine the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are called camera parameters. The process of solving for these parameters is called camera calibration.

[0055] In the context of autonomous driving, this invention utilizes camera sensors to reconstruct spatial 3D information, primarily employing binocular or stereo cameras with more intuitive computational principles. The basic principle involves using stereo matching algorithms to calculate the disparity between cameras, thereby estimating spatial information. Stereo matching without any prior constraints results in extremely high computational costs. To reduce computational burden, epipolar constraint models are generally used to determine the geometric relationships between cameras. This invention focuses on the secondary calibration of binocular or stereo cameras subject to epipolar constraints. The purpose of secondary calibration is to ensure epipolar constraints between cameras while improving the accuracy of reconstructed spatial 3D information. To achieve these objectives, this invention provides a binocular camera secondary calibration method. This method uses a small, portable target, is suitable for outdoor secondary calibration, has low operational complexity, and provides high accuracy in the rotation and translation matrices.

[0056] Please refer to Figure 1 , Figure 1 This is one of the flowcharts for the binocular camera secondary calibration method provided by the present invention.

[0057] In one specific embodiment, the binocular camera secondary calibration method provided by the present invention includes the following steps:

[0058] S110: Two target objects are photographed using the left eye camera of the camera to be calibrated to obtain left images of the two target objects; two target objects are also photographed using the right eye camera of the camera to be calibrated to obtain right images of the two target objects. That is, two target objects, a first target and a second target, are provided. The first target and the second target are placed in different shooting positions. The left eye camera photographs the first target to obtain a left image, and the left eye camera photographs the second target to obtain another left image; the right eye camera photographs the first target to obtain a right image, and the right eye camera photographs the second target to obtain another right image.

[0059] S120: Extract the left target corner points of the two target objects in the left image, and extract the right target corner points of the two target objects in the right image. Since the left image was taken from two target objects separately, the left target corner points should also be in two sets, one set based on the left target corner point of the first target and the other set based on the left target corner point of the second target; similarly, since the right image was taken from two target objects separately, the right target corner points should also be in two sets, one set based on the right target corner point of the first target and the other set based on the right target corner point of the second target.

[0060] S130: Extract the left vanishing point in the left image based on the left target corner point of the two target objects, and extract the right vanishing point in the right image based on the right target corner point of the two target objects;

[0061] S140: Based on the left vanishing point and the right vanishing point, calculate the pitch angle and yaw angle of the rotation matrix between the left eye camera and the right eye camera;

[0062] S150: Calculate the left and right three-dimensional coordinates of the target corner point based on the monocular principle. Specifically, the left three-dimensional coordinates of the target corner point are calculated using a first preset formula, and the right three-dimensional coordinates of the target corner point are calculated using a second preset formula;

[0063] The first preset formula is:

[0064] In the formula, L1 is the three-dimensional spatial distance between the top corners of the two target points in the left image, and P1 is the image distance between the top corners of the two target points in the left image. The three-dimensional distance between the target corner points in the left image is... The left three-dimensional coordinates of the target corner point, and F1 represents the focal length of the left camera;

[0065] The second preset formula is:

[0066] In the formula, L2 is the three-dimensional spatial distance between the top corners of the two target points in the right figure, and P2 is the image distance between the top corners of the two target points in the right figure. The three-dimensional distance is the target corner point in the right figure. F1 represents the right three-dimensional coordinates of the target corner point, and F2 represents the focal length of the right camera.

[0067] S160: Calculate the translation matrix between the left and right cameras based on the left three-dimensional coordinates and the right three-dimensional coordinates.

[0068] In principle, based on the three-dimensional spatial calculation principle of epipolar constrained binocular cameras, the farther the object is from the lens, the smaller the parallax value, as shown in Formula 1.

[0069]

[0070] Where Z is the spatial distance, B is the three-dimensional spatial distance between the center points of the two cameras, F is the pixel focal length of the camera, and D is the pixel difference, i.e., the parallax value, of the three-dimensional spatial point mapped on different camera planes. When the parallax value is 0, the three-dimensional spatial distance Z is infinite, which represents the infinite distance from the camera.

[0071] According to the principle of monocular perspective, two parallel lines on any plane in three-dimensional space intersect at a single point on the image, which is the vanishing point. The vanishing point corresponds to a position on the parallel lines at an infinite distance from the camera. For example... Figure 2 As shown, on two straight lines parallel to the optical center of the image, the point farther away from the camera intersects the camera plane and is closer to the center of the image. When the distance is infinite, it is the center of the image, that is, the coordinates of the vanishing point are the center of the image.

[0072] Assuming the intersection of the optical center line and the camera plane in the camera coordinate system is the image center point with coordinates (0,0), and the point at infinity on a line parallel to the optical center line is (X,Y,Z), after the camera coordinate system is rotated in space sequentially by pitch angle α, yaw angle β, and roll angle θ, what are the coordinates of the new camera coordinate system after the rotation? (X,Y,Z) satisfies the relationship shown in formula (2):

[0073]

[0074] Furthermore, since the image coordinates (u,v,1) and world coordinates in the camera coordinate system satisfy the relationship shown in formula (3):

[0075] Where s is the scaling factor. This represents the focal length of the camera on the X and Y axes.

[0076] Based on formulas (2) and (3), it can be seen that in monocular vision, if the vanishing point of any parallel line on the spatial plane is known, the pitch and yaw angles of the camera coordinates relative to the plane can be calculated. Similarly, for binocular cameras, the vanishing points of the points at infinity of the parallel lines in the same space between the two cameras on the plane of the two cameras can be calculated to determine the pitch and yaw angles of the rotation matrices of the two cameras.

[0077] The roll angle of the rotation matrix of a binocular camera can be directly calculated from the angle between the mapping lines of the three-dimensional spatial lines of the two parallel cameras on the image planes of the two cameras.

[0078] At this point, the rotation matrices of the two cameras can be completely solved.

[0079] Based on the principle of binocular cameras using the epipolar constraint model, it is known that, according to the rotation matrix between the cameras, rotating one of the cameras can make the planes of the two cameras parallel, such as... Figure 3 and Figure 4 As shown.

[0080] Assuming the same physical point in space is known, the difference in coordinates between the two cameras can be used to obtain the translation matrix, as shown in formula (5):

[0081] in Let be the translation vector, and let be the 3D coordinates of the same physical point on the left camera. The three-dimensional coordinates of the right camera are .

[0082] Similarly, based on the principle of stereo cameras using the epipolar constraint model, if the rotation and translation matrices of the stereo cameras are known, the epipolar correction method can make the planes of the two cameras concentric and coplanar, such as... Figure 5 As shown.

[0083] As can be seen from the above principle analysis, in order to perform secondary calibration of the epipolar constraint model of a binocular camera, it is necessary to obtain the vanishing point of the same spatial parallel line on different cameras and the three-dimensional coordinate difference of the same physical coordinate point on different cameras.

[0084] In some embodiments, connecting all corresponding target corner points of the two target targets in the left figure with a line, the process further includes:

[0085] Rotate the left image using a preset rotation matrix.

[0086] Further, the pitch and yaw angles of the rotation matrix between the left and right cameras are calculated, followed by the following steps:

[0087] Connecting all corresponding target corner points of the two targets in the left diagram yields the equation of the left straight line. Similarly, connecting all corresponding target corner points of the two targets in the right diagram yields the equation of the right straight line. In the left diagram, the first target has one set of target corner points, and the second target has another set of target corner points. Connecting the corresponding target corner points yields multiple lines connecting the target corner points, from which the equation of the left straight line can be obtained. Likewise, in the right diagram, connecting multiple target corner points yields the equation of the right straight line.

[0088] The roll angle of the rotation matrix is ​​calculated based on the equations of the left and right lines.

[0089] The equation of the left straight line is expressed as follows:

[0090] in, Represents the vertical coordinates in the left image. Represents the horizontal coordinates in the left image. This represents the slope of the line equation in the left figure. This represents the intercept of the equation of the line in the left figure.

[0091] The equation of the right-hand line is expressed as follows:

[0092] in, Represents the vertical coordinates in the right-hand image. This represents the horizontal coordinate in the right image. This represents the slope of the line equation in the right-hand graph. This represents the intercept of the equation of the line in the right figure.

[0093] To facilitate understanding, the implementation process of the binocular camera secondary calibration method provided by this invention will be briefly described below using a specific application scenario as an example. Figure 6 As shown, the method includes the following steps:

[0094] To create lightweight targets, for example, such as... Figure 7 As shown, the number of points at the top corner of the target can be 1×7;

[0095] Images were taken from different locations in space where the target was placed, resulting in two images of the target. Figure 7 and Figure 8 As shown;

[0096] Extract the corner points of the target from the two images;

[0097] Based on two points to determine a straight line, the same corner point of the target at different spatial positions is a straight line in three-dimensional space. The straight lines corresponding to different corner points on the same target are parallel to each other, thus determining the parallel lines.

[0098] According to the principles of perspective, the point where parallel lines intersect on the coordinate system of an image is the vanishing point, such as... Figure 9 As shown;

[0099] The yaw and pitch angles of the rotation matrix between the cameras can be solved based on the vanishing points in the image coordinates of the two cameras.

[0100] Rotate the left image using a rotation matrix;

[0101] The roll angle of the rotation matrix can be directly calculated by using the angle between the lines connecting the target corner points on the images of the two cameras at the same three-dimensional spatial position.

[0102] Based on the monocular principle, the three-dimensional spatial distance of each target corner point can be calculated from the known image coordinates of each pair of target corner points;

[0103] The translation matrix of the two cameras is calculated based on the distance difference between the target corner points and the two cameras.

[0104] This completes the calibration of the rotation and translation matrices between the two cameras, thus completing the secondary calibration.

[0105] In the above specific embodiments, the binocular camera secondary calibration method and apparatus provided by the present invention captures two target images using the left eye camera of the camera to be calibrated, to obtain a left image of the two target images; captures two target images using the right eye camera of the camera to be calibrated, to obtain a right image of the two target images; extracts the left target corner points of the two target images from the left image, and extracts the right target corner points of the two target images from the right image; extracts the left vanishing point in the left image based on the left target corner points of the two target images, and extracts the right vanishing point in the right image based on the right target corner points of the two target images; calculates the pitch and yaw angles of the rotation matrix between the left and right eye cameras based on the left and right vanishing points; calculates the left and right three-dimensional coordinates of the target corner points based on the monocular principle; and calculates the translation matrix between the left and right eye cameras based on the left and right three-dimensional coordinates. This invention utilizes the vanishing point principle of perspective, the image ranging principle, and the multi-view 3D coordinate difference to perform secondary calibration of the epipolar constraint model of a binocular camera. In autonomous driving or assisted driving scenarios, it improves the calculation accuracy of the rotation matrix and translation matrix, thereby improving the accuracy of the secondary calibration of the binocular camera.

[0106] In addition to the methods described above, the present invention also provides a binocular camera secondary calibration device, such as... Figure 10 As shown, the device includes:

[0107] The image acquisition unit 1010 is used to capture two target images through the left eye camera of the camera to be calibrated, so as to obtain a left image of the two target images; and to capture two target images through the right eye camera of the camera to be calibrated, so as to obtain a right image of the two target images.

[0108] The corner point extraction unit 1020 is used to extract the left corner points of the two target targets in the left image and the right corner points of the two target targets in the right image.

[0109] The vanishing point extraction unit 1030 is used to extract the left vanishing point in the left image based on the left target corner point of the two target objects, and to extract the right vanishing point in the right image based on the right target corner point of the two target objects.

[0110] The rotation matrix calibration unit 1040 is used to calculate the pitch angle and yaw angle of the rotation matrix between the left eye camera and the right eye camera based on the left vanishing point and the right vanishing point;

[0111] The 3D coordinate calculation unit 1050 is used to calculate the left and right 3D coordinates of the target corner points based on the monocular principle.

[0112] The translation matrix calibration unit 1060 is used to calculate the translation matrix between the left and right cameras based on the left three-dimensional coordinates and the right three-dimensional coordinates.

[0113] In some embodiments, the pitch and yaw angles of the rotation matrix between the left and right cameras are calculated, followed by:

[0114] Connect all the corresponding target corner points of the two targets in the left figure to obtain the equation of the left straight line; connect all the corresponding target corner points of the two targets in the right figure to obtain the equation of the right straight line.

[0115] The roll angle of the rotation matrix is ​​calculated based on the equations of the left and right lines.

[0116] In some embodiments, connecting all corresponding target corner points of the two target targets in the left figure with a line, the process further includes:

[0117] Rotate the left image using a preset rotation matrix.

[0118] In some embodiments, the expression for the equation of the left straight line is:

[0119] in, Represents the vertical coordinates in the left image. Represents the horizontal coordinates in the left image. This represents the slope of the line equation in the left figure. This represents the intercept of the equation of the line in the left figure.

[0120] In some embodiments, the equation of the right-hand line is expressed as:

[0121] in, Represents the vertical coordinates in the right-hand image. This represents the horizontal coordinate in the right image. This represents the slope of the line equation in the right-hand graph. This represents the intercept of the equation of the line in the right figure.

[0122] In some embodiments, the left three-dimensional coordinates of the target corner point are calculated using a first preset formula, and the right three-dimensional coordinates of the target corner point are calculated using a second preset formula.

[0123] The first preset formula is:

[0124] In the formula, L1 is the three-dimensional spatial distance between the top corners of the two target points in the left image, and P1 is the image distance between the top corners of the two target points in the left image. The three-dimensional distance between the target corner points in the left image is... The left three-dimensional coordinates of the target corner point, and F1 represents the focal length of the left camera;

[0125] The second preset formula is:

[0126] In the formula, L2 is the three-dimensional spatial distance between the top corners of the two target points in the right figure, and P2 is the image distance between the top corners of the two target points in the right figure. The three-dimensional distance is the target corner point in the right figure. F1 represents the right three-dimensional coordinates of the target corner point, and F2 represents the focal length of the right camera.

[0127] In the above specific embodiments, the binocular camera secondary calibration device provided by the present invention captures two target images by the left eye camera of the camera to be calibrated, to obtain a left image of the two target images; captures two target images by the right eye camera of the camera to be calibrated, to obtain a right image of the two target images; extracts the left target corner points of the two target images respectively in the left image, and extracts the right target corner points of the two target images respectively in the right image; extracts the left vanishing point in the left image based on the left target corner points of the two target images, and extracts the right vanishing point in the right image based on the right target corner points of the two target images; calculates the pitch angle and yaw angle of the rotation matrix between the left eye camera and the right eye camera based on the left vanishing point and the right vanishing point; calculates the left three-dimensional coordinates and right three-dimensional coordinates of the target corner points based on the monocular principle; and calculates the translation matrix between the left eye camera and the right eye camera based on the left three-dimensional coordinates and the right three-dimensional coordinates. This invention utilizes the vanishing point principle of perspective, the image ranging principle, and the multi-view 3D coordinate difference to perform secondary calibration of the epipolar constraint model of a binocular camera. In autonomous driving or assisted driving scenarios, it improves the calculation accuracy of the rotation matrix and translation matrix, thereby improving the accuracy of the secondary calibration of the binocular camera.

[0128] The present invention also provides a smart terminal, the smart terminal comprising: a data acquisition device, a processor, and a memory;

[0129] The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions to perform the method described above.

[0130] Corresponding to the above embodiments, this invention also provides a computer-readable storage medium containing one or more program instructions. These one or more program instructions are configured to be executed by a binocular camera depth calibration system using the method described above.

[0131] The present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to perform the methods described above.

[0132] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0133] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0134] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0135] Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0136] Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM).

[0137] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0138] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using a combination of hardware and software. When applied as software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer.

[0139] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for secondary calibration of a binocular camera, characterized in that, The method includes: Two target images are captured by the left eye camera of the camera to be calibrated to obtain a left image of the two target images; two target images are captured by the right eye camera of the camera to be calibrated to obtain a right image of the two target images. In the left image, extract the left corner points of the two target objects respectively; in the right image, extract the right corner points of the two target objects respectively. Extract the left vanishing point in the left image based on the left corner of the two target objects, and extract the right vanishing point in the right image based on the right corner of the two target objects; Based on the left vanishing point and the right vanishing point, calculate the pitch angle and yaw angle of the rotation matrix between the left eye camera and the right eye camera; The left three-dimensional coordinates of the left target corner and the right three-dimensional coordinates of the right target corner are calculated based on the monocular principle. Calculate the translation matrix between the left and right cameras based on the left and right 3D coordinates; wherein, calculate the pitch and yaw angles of the rotation matrix between the left and right cameras, followed by: Connect all the corresponding target corner points of the two targets in the left figure to obtain the equation of the left straight line; connect all the corresponding target corner points of the two targets in the right figure to obtain the equation of the right straight line. Calculate the roll angle of the rotation matrix based on the left and right line equations; The process of connecting all corresponding target corner points of the two target targets in the left figure includes, prior to: Rotate the left image using a preset rotation matrix.

2. The binocular camera secondary calibration method according to claim 1, characterized in that, The equation of the left straight line is expressed as follows: in, Represents the vertical coordinates in the left image. Represents the horizontal coordinates in the left image. This represents the slope of the line equation in the left figure. This represents the intercept of the equation of the line in the left figure.

3. The binocular camera secondary calibration method according to claim 1, characterized in that, The equation of the right-hand line is expressed as follows: in, Represents the vertical coordinates in the right-hand image. This represents the horizontal coordinate in the right image. This represents the slope of the line equation in the right-hand graph. This represents the intercept of the equation of the line in the right figure.

4. The binocular camera secondary calibration method according to claim 1, characterized in that, The left three-dimensional coordinates of the left target corner point are calculated using the first preset formula, and the right three-dimensional coordinates of the right target corner point are calculated using the second preset formula. The first preset formula is: In the formula, L1 is the three-dimensional spatial distance between the top corners of the two target points in the left image, and P1 is the image distance between the top corners of the two target points in the left image. The distance is the three-dimensional distance to the left corner of the target in the left image. The left three-dimensional coordinates of the left target corner point, and F1 represents the focal length of the left camera; The second preset formula is: In the formula, L2 is the three-dimensional spatial distance between the top corners of the two target points in the right figure, and P2 is the image distance between the top corners of the two target points in the right figure. The distance to the right target corner in the right figure is 3D. F2 represents the right three-dimensional coordinates of the right target corner point, and F2 represents the right camera focal length.

5. A secondary calibration device for a binocular camera, characterized in that, The device includes: The image acquisition unit is used to capture images of two target objects using the left eye camera of the camera to be calibrated, so as to obtain a left image of the two target objects; and to capture images of the two target objects using the right eye camera of the camera to be calibrated, so as to obtain a right image of the two target objects. The corner point extraction unit is used to extract the left corner points of the two target targets in the left image and the right corner points of the two target targets in the right image. The vanishing point extraction unit is used to extract the left vanishing point in the left image based on the left target corner point of the two target objects, and to extract the right vanishing point in the right image based on the right target corner point of the two target objects. The rotation matrix calibration unit is used to calculate the pitch angle and yaw angle of the rotation matrix between the left eye camera and the right eye camera based on the left vanishing point and the right vanishing point; The three-dimensional coordinate calculation unit is used to calculate the left three-dimensional coordinates of the left target corner point and the right three-dimensional coordinates of the right target corner point based on the monocular principle; The translation matrix calibration unit is used to calculate the translation matrix between the left and right cameras based on the left three-dimensional coordinates and the right three-dimensional coordinates. The calculation includes the pitch and yaw angles of the rotation matrix between the left and right cameras, followed by: Connect all the corresponding target corner points of the two targets in the left figure to obtain the equation of the left straight line; connect all the corresponding target corner points of the two targets in the right figure to obtain the equation of the right straight line. Calculate the roll angle of the rotation matrix based on the left and right line equations; The process of connecting all corresponding target corner points of the two target targets in the left figure includes, prior to: Rotate the left image using a preset rotation matrix.

6. A smart terminal, characterized in that, The intelligent terminal includes: a data acquisition device, a processor, and a memory; The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions to perform the method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains one or more program instructions that, when executed by a processor, implement the method as described in any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-4.

Images