A chessboard-based binocular camera rapid splicing method

Abstract

The application provides a chessboard-based binocular camera rapid splicing method, a binocular camera image splicing based on a chessboard detection technology, and a whole process efficiency several times higher than that of a traditional splicing method by using binocular camera splicing, perspective transformation based on a chessboard vertex, and pixel weighted fusion, so that the whole process can be successfully applied to real-time stream splicing. The chessboard detection method is used to quickly splice photos taken by a binocular camera.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a rapid stitching method for binocular cameras based on a checkerboard pattern. Background Technology

[0002] With the development of technology, existing technologies are increasingly being used in facial liveness recognition systems, especially binocular cameras. Because most binocular cameras on the market use different modules, and due to inherent differences between the two cameras themselves, variations in imaging parameters and assembly precision are inevitable. Current binocular alignment methods perform single-camera alignment for each camera separately. However, existing image stitching methods, such as SURF, SIFT, and ORB, are time-consuming in finding feature points, resulting in low image stitching efficiency and making them unsuitable for real-time streaming stitching with high frame rate requirements.

[0003] In addition, the following are commonly used terms in the prior art:

[0004] 1. Binocular camera stitching: stitching together photos taken by two cameras placed at a certain angle to merge them into a single image.

[0005] 2. Camera Calibration: In image measurement and machine vision applications, to determine the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are the camera parameters. Under most conditions, these parameters must be obtained through experiments and calculations; this process of solving for the parameters is called camera calibration (or camera camera calibration).

[0006] 3. Weighted fusion: Multiply the pixel values ​​of the image by a certain weight and then add them together.

[0007] 4. Corner point: A corner point is an extreme point, that is, a point that is particularly prominent in a certain aspect. It can be the intersection of two lines, or a point located on two adjacent things with different main directions.

[0008] 5. Least squares method: a mathematical tool that is widely used in many disciplines of data processing, such as error estimation, uncertainty, system identification and prediction, forecasting, etc. Summary of the Invention

[0009] In order to solve the problems in the prior art, the purpose of this application is to quickly stitch together photos taken by a binocular camera using a checkerboard detection method.

[0010] Specifically, the present invention provides a method for rapid stitching of images using a binocular camera based on a checkerboard pattern, the method comprising the following steps:

[0011] S1, Start, Initialize settings;

[0012] S2, the left and right cameras acquire real-time frame data respectively;

[0013] S3, read and save the coordinate file of the four vertices of the chessboard grid;

[0014] S4. Determine if the vertex coordinates have been obtained. If not, proceed to step S5 and then return to step S3; if yes, proceed to step S6.

[0015] S5, camera calibration, checkerboard detection;

[0016] S6, Perspective Transformation:

[0017] The coordinates of the four vertices of the chessboard detected by the left and right cameras are weighted, averaged, and scaled to generate new coordinates of the four vertices of the chessboard; the calculation formula is as follows:

[0018]

[0019]

[0020] new_chessboard_y=scale_y×(temp_y-img_height / 2)+stitching_height / 2

[0021] new_chessboard_x=scale_x×(temp_x-img_width / 2)+stitching_width / 2

[0022] Where `new_chessboard_x` and `new_chessboard_y` are the x and y coordinates of the vertices of the new chessboard grid; `left_x` and `left_y` are the x and y coordinates of the vertices of the chessboard grid captured by the left camera; `right_x` and `right_y` are the x and y coordinates of the vertices of the chessboard grid captured by the right camera; `img_width` and `img_height` are the width and height of the images captured by the left and right cameras; `temp_x` and `temp_y` are the x and y coordinates of the intermediate temporary variable point; `scale_x` and `scale_y` are the scaling factors in the x and y directions; `stitching_width` and `stitching_height` are the width and height of the stitched image, respectively, generating the coordinates of the four vertices of the new chessboard grid; the perspective transformation matrix from the left and right camera chessboard grids to the new chessboard grid is calculated respectively, so that the left and right images are mapped onto one image, i.e., stitched onto one image. The calculation formula is as follows:

[0023]

[0024]

[0025] in , Let a be the coordinate of the i-th vertex among the four vertices of the new chessboard. ij These are the values ​​of the j-th element in the i-th row of a 3×3 perspective transformation matrix, where a 33 x is a constant value of 1, and the rest are values ​​to be determined; i y i Let be the coordinates of the i-th vertex among the four vertices of the left / right chessboard; solve for 'a' based on the coordinates of the four vertices of the left / right chessboard and the coordinates of the four vertices of the new chessboard. ij The value is used to obtain the perspective transformation matrix that maps the left / right images to the stitched image;

[0026] S7, Weighted Fusion:

[0027] The stitched image contains gaps with uneven brightness transitions. To address this, a weighted average is applied to the 20 pixels to the left of the gap to achieve a more uniform brightness transition. The calculation formula is as follows:

[0028]

[0029] pixel i The pixel value of the i-th pixel from left to right within a 20-pixel region to the left of the gap; pixel_new i The pixel value is the weighted average of the i-th pixel; pixel_border is the pixel value of the pixel in the gap; length i Let be the distance from the i-th pixel to the seam of the stitching;

[0030] S8, output the mosaic image;

[0031] S9, End.

[0032] Step S5 further includes:

[0033] S5.1, Camera Calibration: Calibrate the stereo cameras separately to obtain the camera's intrinsic parameter matrix and distortion parameters. Camera calibration only needs to be performed once and is part of the data preparation before image stitching.

[0034] S5.2, Checkerboard Detection: Before image stitching from the binocular camera, the corner coordinates of the checkerboard in the left / right images need to be detected; the average width of each square in the checkerboard is calculated using the X-coordinates of the corners; the X-coordinates of the uncaptured checkerboard vertices are inferred based on the number of checkerboard squares and the X-coordinates of the corners; the detected corner coordinates are fitted into a straight line equation using the least squares method; the Y-coordinates of the uncaptured checkerboard vertices are inferred using this straight line equation and the calculated X-coordinates of the uncaptured checkerboard vertices; the coordinates of the four checkerboard vertices are saved to a file, and the checkerboard coordinates are no longer detected subsequently, only the file needs to be read.

[0035] Step S6 further includes:

[0036] S6.1 The left and right cameras respectively capture the chessboard grid. The left camera obtains the positions of the top left and bottom left vertices of the left half of the chessboard grid and infers the positions of the top right and bottom right vertices of the chessboard grid. The right camera obtains the positions of the top right and bottom right vertices of the right half of the chessboard grid and infers the positions of the top left and bottom left vertices of the chessboard grid.

[0037] S6.2, by using the positions of the top left and bottom left vertices of the chessboard grid in the left image and the predicted positions of the top left and bottom left vertices, and the positions of the top right and bottom right vertices of the chessboard grid in the right image and the predicted positions of the top right and bottom right vertices, calculate the positions of the top left, bottom left, top right, and bottom right vertices of the chessboard grid in the stitched image.

[0038] S6.3, based on the results of S6.2, obtain the four vertices of the chessboard grid in the spliced ​​image.

[0039] Therefore, the advantage of this application lies in the fact that this method improves the efficiency of the entire process by several times compared to traditional stitching methods, and can be successfully applied to real-time stream stitching. It utilizes a checkerboard detection method to quickly stitch together images captured by a binocular camera. Attached Figure Description

[0040] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0041] Figure 1 This is a flowchart of the method of the present invention.

[0042] Figure 2 This is a schematic diagram of binocular camera shooting involved in the method of the present invention.

[0043] Figure 3 This is a schematic diagram of the left camera image in an embodiment of the method of the present invention.

[0044] Figure 4 This is a schematic diagram of the right camera image in an embodiment of the method of the present invention.

[0045] Figure 5 This is a schematic diagram illustrating the perspective transformation principle involved in the method of this invention.

[0046] Figure 6 This is a schematic diagram of the stitched image in an embodiment of the method of the present invention.

[0047] Figure 7 This is a schematic diagram of the optimized image in an embodiment of the method of the present invention. Detailed Implementation

[0048] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0049] like Figure 1 As shown, the method of the present invention relates to a rapid stitching method for binocular cameras based on a checkerboard pattern, the method comprising the following steps:

[0050] S1, Start, Initialize settings;

[0051] S2, the left and right cameras acquire real-time frame data respectively;

[0052] S3, read and save the coordinate file of the four vertices of the chessboard grid;

[0053] S4. Determine if the vertex coordinates have been obtained. If not, proceed to step S5 and then return to step S3; if yes, proceed to step S6.

[0054] S5, camera calibration, checkerboard detection;

[0055] S6, Perspective Transformation:

[0056] The coordinates of the four vertices of the chessboard detected by the left and right cameras are weighted, averaged, and scaled to generate new coordinates of the four vertices of the chessboard; the calculation formula is as follows:

[0057]

[0058]

[0059] new_chessboard_y=scale_y×(temp_y-img_height / 2)+stitching_height / 2

[0060] new_chessboard_x=scale_x×(temp_x-img_width / 2)+stitching_width / 2

[0061] Where `new_chessboard_x` and `new_chessboard_y` are the x and y coordinates of the vertices of the new chessboard grid; `left_x` and `left_y` are the x and y coordinates of the vertices of the chessboard grid captured by the left camera; `right_x` and `right_y` are the x and y coordinates of the vertices of the chessboard grid captured by the right camera; `img_width` and `img_height` are the width and height of the images captured by the left and right cameras; `temp_x` and `temp_y` are the x and y coordinates of the intermediate temporary variable point; `scale_x` and `scale_y` are the scaling factors in the x and y directions; `stitching_width` and `stitching_height` are the width and height of the stitched image, respectively, generating the coordinates of the four vertices of the new chessboard grid; the perspective transformation matrix from the left and right camera chessboard grids to the new chessboard grid is calculated respectively, so that the left and right images are mapped onto one image, i.e., stitched onto one image. The calculation formula is as follows:

[0062]

[0063]

[0064] in , Let a be the coordinate of the i-th vertex among the four vertices of the new chessboard. ij These are the values ​​of the j-th element in the i-th row of a 3×3 perspective transformation matrix, where a 33 x is a constant value of 1, and the rest are values ​​to be determined; i y i Let be the coordinates of the i-th vertex among the four vertices of the left / right chessboard; solve for 'a' based on the coordinates of the four vertices of the left / right chessboard and the coordinates of the four vertices of the new chessboard. ij The value is used to obtain the perspective transformation matrix that maps the left / right images to the stitched image;

[0065] S7, Weighted Fusion:

[0066] The stitched image contains gaps with uneven brightness transitions. To address this, a weighted average is applied to the 20 pixels to the left of the gap to achieve a more uniform brightness transition. The calculation formula is as follows:

[0067]

[0068] pixel i The pixel value of the i-th pixel from left to right within a 20-pixel region to the left of the gap; pixel_new i The pixel value is the weighted average of the i-th pixel; pixel_border is the pixel value of the pixel in the gap; length i Let be the distance from the i-th pixel to the seam of the stitching;

[0069] S8, output the mosaic image;

[0070] S9, End.

[0071] The camera calibration involves calibrating the stereo cameras separately to obtain their intrinsic parameter matrices and distortion parameters. Camera calibration only needs to be performed once and is part of the data preparation before image stitching.

[0072] Among these steps, checkerboard detection involves detecting the vertex coordinates of the checkerboard grid in the image before image stitching using a stereo camera, and inferring the vertex coordinates of the missing checkerboard grid segments. A diagram illustrating a stereo camera's checkerboard image capture is shown below. Figure 2 As shown in the image. Actual photographs are as follows. Figure 3 and Figure 4 As shown, the coordinates of the four vertices of the chessboard are saved to a file. Subsequent checks of the chessboard coordinates are no longer necessary; only the file needs to be read. This step greatly improves the stitching efficiency.

[0073] The perspective transformation involves weighted averaging and scaling the coordinates of the four vertices of the checkerboard detected by the left and right cameras to generate new checkerboard coordinates. The perspective transformation matrices from the left and right camera checkerboards to the new checkerboard are calculated separately, mapping the left and right images onto a single image. The principle is as follows: Figure 5 As shown. The splicing effect is as follows. Figure 6 As shown.

[0074] In step 3, weighted fusion addresses the issue of uneven brightness gaps in the stitched images. To resolve this, a weighted average is applied to the pixels at these gaps, resulting in a more uniform brightness transition. Figure 7 As shown.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A rapid stitching method for binocular cameras based on a checkerboard pattern, characterized in that, The method includes the following steps: S1, Start, Initialize settings; S2, the left and right cameras acquire real-time frame data respectively; S3, read the saved coordinate file of the four vertices of the chessboard grid; S4. Determine if the vertex coordinates have been obtained. If not, proceed to step S5 and then return to step S3; if yes, proceed to step S6. S5, camera calibration, checkerboard detection; S6, Perspective Transformation: The coordinates of the four vertices of the chessboard detected by the left and right cameras are weighted, averaged, and scaled to generate new coordinates of the four vertices of the chessboard; the calculation formula is as follows: ； ； new_chessboard_y=scale_y×(temp_y-img_height / 2)+stitching_height / 2; new_chessboard_x=scale_x×(temp_x-img_width / 2)+stitching_width / 2; Where `new_chessboard_x` and `new_chessboard_y` are the x and y coordinates of the vertices of the new chessboard grid; `left_x` and `left_y` are the x and y coordinates of the vertices of the chessboard grid captured by the left camera; `right_x` and `right_y` are the x and y coordinates of the vertices of the chessboard grid captured by the right camera; `img_width` and `img_height` are the width and height of the images captured by the left and right cameras; `temp_x` and `temp_y` are the x and y coordinates of the intermediate temporary variable point; `scale_x` and `scale_y` are the scaling factors in the x and y directions; `stitching_width` and `stitching_height` are the width and height of the stitched image, respectively; the perspective transformation matrix from the left and right camera chessboard grids to the new chessboard grid is calculated separately, so that the left and right images are mapped onto one image, i.e., stitched onto one image. The calculation formula is as follows: ； ； in , Let a be the coordinate of the i-th vertex among the four vertices of the new chessboard. ij These are the values ​​of the j-th element in the i-th row of a 3×3 perspective transformation matrix, where a 33 x is a constant value of 1, and the rest are values ​​to be determined; i y i Let be the coordinates of the i-th vertex among the four vertices of the left / right chessboard; solve for 'a' based on the coordinates of the four vertices of the left / right chessboard and the coordinates of the four vertices of the new chessboard. ij The value is used to obtain the perspective transformation matrix that maps the left / right images to the stitched image; S7, Weighted Fusion: The stitched image contains gaps with uneven brightness transitions. To address this, a weighted average is applied to the 20 pixels to the left of the gap to achieve a more uniform brightness transition. The calculation formula is as follows: ； pixel i The pixel value of the i-th pixel from left to right within a 20-pixel region to the left of the gap; pixel_new i The pixel value is the weighted average of the i-th pixel; pixel_border is the pixel value of the pixel in the gap; length i Let be the distance from the i-th pixel to the seam of the stitching; S8, output the mosaic image; S9, End.

2. The method for rapid stitching of stereo cameras based on a checkerboard pattern according to claim 1, characterized in that, Step S5 further includes: S5.1, Camera Calibration: Calibrate the stereo cameras separately to obtain the camera's intrinsic parameter matrix and distortion parameters. Camera calibration only needs to be performed once and is part of the data preparation before image stitching. S5.2, Checkerboard Detection: Before image stitching from the binocular camera, the corner coordinates of the checkerboard in the left / right images need to be detected; the average width of each square in the checkerboard is calculated using the X-coordinates of the corners; the X-coordinates of the uncaptured checkerboard vertices are inferred based on the number of checkerboard squares and the X-coordinates of the corners; the detected corner coordinates are fitted into a straight line equation using the least squares method; the Y-coordinates of the uncaptured checkerboard vertices are inferred using this straight line equation and the calculated X-coordinates of the uncaptured checkerboard vertices; the coordinates of the four checkerboard vertices are saved to a file, and the checkerboard coordinates are no longer detected subsequently, only the file needs to be read.

3. The method for rapid stitching of stereo cameras based on a checkerboard pattern according to claim 1, characterized in that, Step S6 further includes: S6.1 The left and right cameras respectively capture the chessboard grid. The left camera obtains the positions of the top left and bottom left vertices of the left half of the chessboard grid and infers the positions of the top right and bottom right vertices of the chessboard grid. The right camera obtains the positions of the top right and bottom right vertices of the right half of the chessboard grid and infers the positions of the top left and bottom left vertices of the chessboard grid. S6.2, by using the positions of the top left and bottom left vertices of the chessboard grid in the left image and the predicted positions of the top left and bottom left vertices, and the positions of the top right and bottom right vertices of the chessboard grid in the right image and the predicted positions of the top right and bottom right vertices, calculate the positions of the top left, bottom left, top right, and bottom right vertices of the chessboard grid in the stitched image. S6.3, based on the results of S6.2, obtain the four vertices of the chessboard grid in the spliced ​​image.

Images