A 3D camera system real-time video processing and stereoscopic sense adjustment method and system

Abstract

The real-time video processing and stereoscopic sense adjustment method and system for a 3D camera system disclosed by the present invention include: calibrating a binocular camera using the Zhang Dingyou calibration method according to a checkerboard image to obtain an internal parameter matrix, distortion parameters, and an external parameter matrix; obtaining a corrected rotation matrix of the binocular camera according to the external parameter matrix; performing stereo reprojection on two frames of images captured by the binocular camera to obtain the projected image coordinates; obtaining a vertically corrected parallax image according to the corrected rotation matrix, internal parameter matrix, distortion parameters, and projected image coordinates of the binocular camera; calculating a horizontal parallax statistical value according to the vertically corrected parallax image; and adjusting the horizontal parallax according to the horizontal parallax statistical value to obtain an adjusted 3D video image. By calibrating the binocular camera, correcting the vertical and horizontal parallax, and performing pixel adjustment on the left and right images of the binocular camera according to the parameters of the display and the distance between the observer and the display, the present invention enables the observer to improve the comfort of video viewing.

Description

Technical Field

[0001] The present invention relates to the field of 3D imaging system calibration and real-time video processing, and particularly to a method and system for real-time video processing and stereoscopic sense adjustment of a 3D camera system. Background Art

[0002] 3D display systems are generally divided into three types: three-dimensional display based on stereoscopic image pairs, holographic three-dimensional display, and volumetric three-dimensional display. As Figure 1 shown, the stereoscopic image display of a 3D camera system is generally based on the principle of three-dimensional display of stereoscopic image pairs. The left and right cameras at a certain distance capture the same target to form left and right images with a certain horizontal pixel difference in the target position; as Figure 2 shown, the left and right images are displayed on a display. The targets with a certain parallax are respectively viewed by the left and right eyes of a person and fused to form a stereoscopic image with corresponding depth information. 3D camera systems are widely used, such as the camera system for 3D movie shooting, the 3D endoscope camera system applied to minimally invasive surgery, etc. During the minimally invasive surgery process, doctors watch the 3D images of human tissues in real time through the 3D endoscope camera system for surgical operations. Therefore, high requirements are imposed on the image processing speed and the comfort of video viewing.

[0003] Generally, the two image sensors of a 3D endoscope camera system are arranged in parallel at a short distance. However, due to insufficient installation process accuracy, the two sensor target surfaces are not completely parallel, and the lenses all have different degrees of distortion, resulting in a vertical parallax of the same target in the left and right images on the display for the human eye, which is not easy to focus. In addition, the 3D endoscope shooting distance is generally relatively short, and a large parallax often occurs, making the human eye unable to focus. Prolonged viewing is likely to cause a sense of dizziness. Moreover, different display devices and viewing distances affect the size of the viewing image parallax, and it is necessary to adjust the image parallax according to different distances and display sizes. Otherwise, the comfort of video viewing is poor. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for real-time video processing and stereoscopic sense adjustment of a 3D camera system, which can overcome the influence of lens distortion, vertical parallax, horizontal parallax, and 3D display size and viewing distance on video images and improve the comfort of video viewing.

[0005] To achieve the above purpose, the present invention provides the following solutions:

[0006] A method for real-time video processing and stereoscopic sense adjustment of a 3D camera system, the method comprising:

[0007] Calibrating a binocular camera according to a checkerboard image by using the Zhang Dingyou calibration method to obtain an internal parameter matrix, distortion parameters, and an external parameter matrix; the binocular camera includes a left camera and a right camera;

[0008] Based on the external parameter matrix, obtain the calibration rotation matrix of the binocular camera;

[0009] Perform stereo reprojection on two frames of images captured by the binocular camera to obtain the projected image coordinates;

[0010] Based on the calibration rotation matrix of the binocular camera, the internal parameter matrix, the distortion parameters, and the projected image coordinates, obtain the vertical parallax corrected image;

[0011] Calculate the horizontal parallax statistical value according to the vertical parallax corrected image;

[0012] Adjust the horizontal parallax according to the horizontal parallax statistical value to obtain the adjusted 3D video image.

[0013] Optionally, the obtaining the calibration rotation matrix of the binocular camera according to the external parameter matrix specifically includes:

[0014] Calculate the camera coordinate system rotation matrix and translation vector of the left camera relative to the right camera according to the external parameter matrix;

[0015] Calculate the mean value of the rotation matrix based on the geodesic distance according to the rotation matrix to obtain the rotation matrix statistical value;

[0016] Calculate the Euler space mean value according to the translation vector to obtain the translation vector statistical value;

[0017] Use the Bouguet algorithm according to the rotation matrix statistical value and the translation vector statistical value to obtain the calibration rotation matrix of the binocular camera.

[0018] Optionally, the obtaining the vertical parallax corrected image according to the calibration rotation matrix of the binocular camera, the internal parameter matrix, the distortion parameters, and the projected image coordinates specifically includes:

[0019] Multiply the projected image coordinates by the inverse matrix of the internal parameter matrix to obtain the binocular camera coordinates;

[0020] Multiply the binocular camera coordinates by the inverse matrix of the calibration rotation matrix of the binocular camera to obtain the binocular camera coordinates before stereo calibration;

[0021] Normalize the binocular camera coordinates before stereo calibration to obtain the normalized coordinates;

[0022] Add the normalized coordinates to the distortion parameters to obtain the original binocular camera coordinates;

[0023] Multiply the original binocular camera coordinates by the internal parameter matrix to obtain the original image coordinates;

[0024] Interpolate the corrected image coordinates according to the original image coordinates to obtain the interpolated image coordinates;

[0025] Remove the coordinates with pixel values less than 0 in the interpolated image coordinates to obtain a cropped image; the aspect ratio of the cropped image is the same as that of the two frames of images captured by the binocular camera;

[0026] Enlarge the size of the cropped image proportionally to the same size as the two frames of images captured by the binocular camera to obtain the vertical parallax corrected image.

[0027] Optionally, the calculating the horizontal parallax statistical value according to the vertical parallax corrected image specifically includes:

[0028] Calculate a horizontal parallax image according to the vertical parallax corrected image;

[0029] Perform a parallax geometric position weighting calculation on the horizontal parallax image to obtain the horizontal parallax statistical value.

[0030] Optionally, the adjusting the horizontal parallax according to the horizontal parallax statistical value to obtain an adjusted 3D video image specifically includes:

[0031] When , the left and right images of the vertical parallax corrected image are respectively moved left and right by pixels to obtain an adjusted 3D video image;

[0032] When , the left and right images of the vertical parallax corrected image are respectively moved right and left by to obtain an adjusted 3D video image;

[0033] When , the left and right images of the vertical parallax corrected image are not adjusted at all to obtain an adjusted 3D video image;

[0034] wherein, η = 2.907×10 4 rad, is a constant, S is the distance between the observer and the display, D is the observer's pupil diameter, Ee is the distance between the observer's two pupils, P w is the width of a unit pixel of the display, Diff mean is the horizontal parallax statistical value, and s is the minimum parallax value.

[0035] A 3D camera system real-time video processing and stereoscopic sense adjustment system, the system includes:

[0036] A calibration module, configured to calibrate the binocular camera according to the checkerboard image by using the Zhang Dingyou calibration method to obtain an internal parameter matrix, distortion parameters, and an external parameter matrix; the binocular camera includes a left camera and a right camera;

[0037] The calibration rotation matrix determination module is used to obtain the calibration rotation matrix of the binocular camera according to the external parameter matrix;

[0038] The reprojection module is used to perform stereo reprojection on two frames of images captured by the binocular camera to obtain the image coordinates after projection;

[0039] The vertical parallax correction image determination module is used to obtain the vertical parallax correction image according to the calibration rotation matrix of the binocular camera, the internal parameter matrix, the distortion parameters, and the image coordinates after projection;

[0040] The horizontal parallax statistical value determination module is used to calculate the horizontal parallax statistical value according to the vertical parallax correction image;

[0041] The adjustment module is used to adjust the horizontal parallax according to the horizontal parallax statistical value to obtain the adjusted 3D video image.

[0042] Optionally, the calibration rotation matrix determination module includes:

[0043] The camera coordinate system rotation matrix and translation vector determination unit is used to calculate the camera coordinate system rotation matrix and translation vector of the left camera relative to the right camera according to the external parameter matrix;

[0044] The rotation matrix statistical value determination unit is used to calculate the mean value of the rotation matrix based on the geodesic distance according to the rotation matrix to obtain the rotation matrix statistical value;

[0045] The translation vector statistical value determination unit is used to calculate the Euler space mean value according to the translation vector to obtain the translation vector statistical value

[0046] The calibration rotation matrix determination unit of the binocular camera is used to obtain the calibration rotation matrix of the binocular camera by using the Bouguet algorithm according to the rotation matrix statistical value and the translation vector statistical value.

[0047] Optionally, the vertical parallax correction image determination module includes:

[0048] The binocular camera coordinate determination unit is used to multiply the image coordinates after projection by the inverse matrix of the internal parameter matrix to obtain the binocular camera coordinates;

[0049] The binocular camera coordinates before stereo calibration determination unit is used to multiply the binocular camera coordinates by the inverse matrix of the calibration rotation matrix of the binocular camera to obtain the binocular camera coordinates before stereo calibration;

[0050] The normalization unit is used to normalize the binocular camera coordinates before stereo calibration to obtain the normalized coordinates;

[0051] The original binocular camera coordinate determination unit is used to add the normalized coordinates and the distortion parameters to obtain the original binocular camera coordinates;

[0052] The original image coordinate determination unit is used to multiply the original binocular camera coordinates by the internal parameter matrix to obtain the original image coordinates;

[0053] The interpolated image coordinate determination unit is used to interpolate the corrected image coordinates according to the original image coordinates to obtain the interpolated image coordinates;

[0054] The cropping unit is used to remove the coordinates with pixel values less than 0 in the interpolated image coordinates to obtain a cropped image; the aspect ratio of the cropped image is the same as the aspect ratio of the two frames of images captured by the binocular camera;

[0055] The vertical parallax correction image determination unit is used to scale up the size of the cropped image proportionally to the same size as the two frames of images captured by the binocular camera to obtain the vertical parallax correction image.

[0056] Optionally, the horizontal parallax statistical value determination module includes:

[0057] The horizontal parallax image determination unit is used to calculate the horizontal parallax image according to the vertical parallax correction image;

[0058] The horizontal parallax statistical value determination unit is used to perform weighted calculation of the parallax geometric position on the horizontal parallax image to obtain the horizontal parallax statistical value.

[0059] Optionally, the adjustment module includes:

[0060] The first adjustment unit is used to, when , the left and right images of the vertical parallax correction image are respectively moved left and right by pixels to obtain an adjusted 3D video image;

[0061] The second adjustment unit is used to, when , the left and right images of the vertical parallax correction image are respectively moved right and left by to obtain an adjusted 3D video image;

[0062] The third adjustment unit is used to, when , the left and right images of the vertical parallax correction image are not adjusted at all to obtain an adjusted 3D video image;

[0063] where η = 2.907×10 4 rad, is a constant, S is the distance between the observer and the display, D is the pupil diameter of the observer, Ee is the distance between the two pupils of the observer, P wis the unit pixel width of the display, Diff mean is the horizontal parallax statistical value, and s is the minimum parallax value.

[0064] According to the specific embodiments provided by the present invention, the following technical effects are disclosed:

[0065] The real-time video processing and stereoscopic sense adjustment method and system for a 3D camera system provided by the present invention include: calibrating a binocular camera using the Zhang Dingyou calibration method according to a checkerboard image to obtain an internal parameter matrix, distortion parameters, and an external parameter matrix; obtaining a corrected rotation matrix of the binocular camera according to the external parameter matrix; performing stereo reprojection on two frames of images captured by the binocular camera to obtain the projected image coordinates; obtaining a vertically corrected parallax image according to the corrected rotation matrix, internal parameter matrix, distortion parameters, and projected image coordinates of the binocular camera; calculating a horizontal parallax statistical value according to the vertically corrected parallax image; and adjusting the horizontal parallax according to the horizontal parallax statistical value to obtain an adjusted 3D video image. By calibrating the binocular camera, performing vertical and horizontal parallax corrections, and finally adjusting the pixels of the left and right images of the binocular camera according to the parameters of the display and the distance between the observer and the display, the present invention enables the observer to overcome the influence of lens distortion, vertical parallax, horizontal parallax, and the size and viewing distance of the 3D display on the video image, improving the comfort of video viewing. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0067] Figure 1 is a schematic diagram of the parallax principle of binocular camera stereo imaging applied in the present invention;

[0068] Figure 2 is a schematic diagram of stereoscopic display applied in the present invention;

[0069] Figure 3 is a flowchart of the real-time video processing and stereoscopic sense adjustment method for the 3D camera system of the present invention;

[0070] Figure 4 is a flowchart of the algorithm of the present invention;

[0071] Figure 5 is a module diagram of the 3D camera system real-time video processing and stereoscopic sense adjustment system of the present invention.

[0072] Symbol Explanation:

[0073] Calibration module - 1, correction rotation matrix determination module - 2, reprojection module - 3, vertical parallax correction image determination module - 4, horizontal parallax statistical value determination module - 5, adjustment module - 6. Detailed implementation mode

[0074] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0075] The purpose of the present invention is to provide a real-time video processing and stereoscopic sense adjustment method and system for a 3D camera system, which can overcome the influence of lens distortion, vertical parallax, horizontal parallax, and the size and viewing distance of a 3D display on video images, and improve the comfort of video viewing.

[0076] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation modes.

[0077] As Figure 3 and Figure 4 shown, the real-time video processing and stereoscopic sense adjustment method for a 3D camera system provided by the present invention includes:

[0078] Step S1: According to the checkerboard image, calibrate the binocular camera using the Zhang Dingyou calibration method to obtain the internal parameter matrix, distortion parameters, and external parameter matrix; the binocular camera includes a left camera and a right camera.

[0079] Specifically, calibrate the left camera and the right camera of the binocular camera respectively. Use the Zhang Zhengyou calibration method to calibrate the left and right cameras of the 3D camera system respectively to obtain the internal parameter matrix, distortion parameters, and external parameter matrix of the left camera and the right camera.

[0080] The calibration process uses N sets of captured checkerboard images. After calibration, the internal parameter matrices of the left and right cameras are respectively:

[0081]

[0082] After calibration, the radial distortion parameters of the left and right cameras are respectively:

[0083]

[0084] Among them, are respectively the 2nd, 4th, and 6th order radial distortion coefficients of the left camera, are respectively the 2nd, 4th, and 6th order radial distortion coefficients of the right camera, α (l), β (l) , γ (l) , are the horizontal scale factor, vertical scale factor, skewness of both horizontal and vertical coordinates, horizontal camera principal point, and vertical camera principal point of the left camera; α (r) , β (r) , γ (r) , are the horizontal scale factor, vertical scale factor, skewness of both horizontal and vertical coordinates, horizontal camera principal point, and vertical camera principal point of the right camera.

[0085] Step S2: Obtain the calibration rotation matrix of the binocular camera according to the external parameter matrix.

[0086] Among them, S2 specifically includes:

[0087] Step S21: Calculate the camera coordinate system rotation matrix and translation vector of the left camera relative to the right camera according to the external parameter matrix.

[0088] Furthermore, calculate the camera coordinate system rotation matrix and translation vector of the left camera relative to the right camera according to each set of calculated external parameter matrices (including one external parameter matrix for each of the left and right cameras):

[0089] The rotation matrix is and the translation vector is

[0090] where are the rotation matrices of the right camera and left camera of the i-th group respectively, are the translation vectors of the right camera and left camera of the i-th group respectively.

[0091] Step S22; Calculate the mean value of the rotation matrix based on the geodesic distance to obtain the rotation matrix statistical value.

[0092] Specifically, after obtaining N groups of rotation matrices, due to calculation and shooting errors, the N groups of rotation matrices are inconsistent. Use the rotation matrix L 2 mean value as the statistical value R of the N groups of rotation matrices. The rotation matrix obtained by this method is more reasonable than the rotation matrix directly obtained by averaging Euler angles.

[0093] Furthermore, calculate the mean value of the rotation matrix L 2 as the statistical value of the N groups of rotation matrices. The specific steps are as follows:

[0094] The first step: Use the Rodrigues transformation to define the transformation [θ, r]: = log(R mat ):

[0095]

[0096] Among them, R mat is the rotation matrix, r = [r x r y r z T is the rotation unit vector corresponding to the rotation matrix R mat , θ is the rotation angle of the rotation matrix, and R mat : = exp([θ, r]) is the log inverse transformation, and the specific formula is as follows:

[0097]

[0098] Step 2: For the N groups of rotation matrices of the left camera First, initialize Set a relatively small tolerance parameter ε > 0 and the maximum number of iterations N iterate , and the initial iteration is at this time t = 0, where t is the number of iterations.

[0099] Calculate Among them Among them, R mat T represents the transpose of the R mat matrix.

[0100] Judge: When ||r|| < ε or t ≥ N iterate , return R and stop the iteration, otherwise update R mat : = R mat ·exp([0, r]); the obtained R mat is the rotation matrix statistical value R of the N groups of rotation matrices of the left camera l .

[0101] Step 3: For the N groups of rotation matrices of the right camera First, initialize Set a relatively small tolerance parameter ε > 0 and the maximum number of iterations N iterate , and the initial iteration is at this time t = 0, where t is the number of iterations.

[0102] Calculate Among them Among them, R mat T represents the transpose of the R mat matrix.

[0103] Judge: When ||r|| < ε or t ≥ N iterate , return R and stop the iteration, otherwise update R mat : = R mat ·exp([0, r]); the obtained R mat is the rotation matrix statistical value R of the N groups of rotation matrices of the right camera​r .

[0104] Step 4: Determine the rotation matrix statistic R based on the obtained R l and R r , and determine the rotation matrix statistic R.

[0105] Specifically, R l and R r are the left camera rotation matrix statistic and the right camera rotation matrix statistic of the rotation matrix statistic R, respectively.

[0106] Step S23: Calculate the Euler space mean based on the translation vector to obtain the translation vector statistic.

[0107] Specifically, after obtaining N groups of translation vectors, the translation vector statistic is directly the Euclidean space mean T of the N translation vectors.

[0108] Step S24: Use the Bouguet algorithm based on the rotation matrix statistic and the translation vector statistic to obtain the calibrated rotation matrix of the binocular camera.

[0109] Specifically, due to process problems, the images captured by the left and right sensors of the binocular camera cannot be coplanar or row-aligned. Binocular calibration is required to make the left and right images coplanar and row-aligned. Otherwise, there will be vertical parallax and poor 3D effects.

[0110] Based on the calculated rotation matrix statistic and translation vector statistic, binocular stereo calibration can be performed as follows:

[0111] Based on the rotation matrix statistic R and the translation vector statistic T, the calibrated rotation matrices of the left and right cameras are obtained respectively where e3 = e1 × e2,[[]]END]] is the rotation matrix that rotates half of the angle of the R rotation matrix. represents rotating half of the angle in the reverse direction of the rotation angle of the R rotation matrix.

[0112] The calculation formula of the Bouguet algorithm

[0113] Step S3: Perform stereo reprojection on two frames of images captured by the binocular camera to obtain the reprojected image coordinates.

[0114] Specifically, perform stereo reprojection on two frames of images I l , I r (one frame is obtained by each of the left camera and the right camera) to make the vertical coordinates of the pixels in the left and right images consistent. The main process is to find the original image coordinates through reverse reprojection transformation for each pixel coordinate after projection and perform bilinear interpolation processing to obtain the final reprojected image and the coordinates of the projected image.

[0115] Step S4: Obtain the vertically disparity-corrected image based on the calibration rotation matrix, intrinsic matrix, distortion parameters of the binocular camera, and the coordinates of the projected image.

[0116] Step S4 specifically includes:

[0117] Step S41: Multiply the coordinates of the projected image by the inverse matrix of the intrinsic matrix to obtain the binocular camera coordinates.

[0118] Specifically, the binocular camera coordinates include the left camera coordinates and the right camera coordinates.

[0119] Furthermore, the method for determining the left camera coordinates is specifically described as follows:

[0120] Any coordinate on the projected image where W and H are the width and height of the original image, and the left camera coordinates are obtained through the inverse matrix transformation of the intrinsic matrix:

[0121] Furthermore, the method for determining the right camera coordinates is specifically described as follows:

[0122] Any coordinate on the projected image where W and H are the width and height of the original image, and the right camera coordinates are obtained through the inverse matrix transformation of the intrinsic matrix:

[0123] Step S42: Multiply the binocular camera coordinates by the inverse matrix of the calibration rotation matrix of the binocular camera to obtain the binocular camera coordinates before stereo calibration.

[0124] Specifically, the binocular camera coordinates before stereo calibration include the left camera coordinates before stereo calibration and the right camera coordinates before stereo calibration.

[0125] Furthermore, the camera coordinates of the left camera are inversely rotated to obtain the left camera coordinates before stereo calibration:

[0126] Furthermore, the camera coordinates of the right camera are inversely rotated to obtain the right camera coordinates before stereo calibration:

[0127] Step S43: Normalize the binocular camera coordinates before stereo calibration to obtain the normalized coordinates.

[0128] Specifically, the normalized coordinates include the left camera normalized coordinates and the right camera normalized coordinates.

[0129] Further, normalize the left camera coordinates to obtain the left camera normalized coordinates:

[0130] Further, normalize the right camera coordinates to obtain the right camera normalized coordinates:

[0131] Step S44: Add the normalized coordinates and the distortion parameters to obtain the original binocular camera coordinates.

[0132] Specifically, the original binocular camera coordinates include the original left camera coordinates and the original right camera coordinates.

[0133] Further, the method for determining the original left camera coordinates is as follows:

[0134] Add the left camera normalized coordinates and the distortion parameters to obtain the original left camera coordinates:

[0135]

[0136] Further, the method for determining the original right camera coordinates is as follows:

[0137] Add the right camera normalized coordinates and the distortion parameters to obtain the original right camera coordinates:

[0138]

[0139] Step S45: Multiply the original binocular camera coordinates by the intrinsic matrix to obtain the original image coordinates.

[0140] Specifically, the original image coordinates include the original left camera image coordinates and the original right camera image coordinates.

[0141] Further, the method for determining the original left camera image coordinates is as follows:

[0142] Convert the original left camera coordinates to obtain the original left image coordinates: where u l and v l represent the horizontal and vertical coordinates of the original left image, respectively.

[0143] Further, the method for determining the original right camera image coordinates is as follows:

[0144] Convert the original right camera coordinates to obtain the original right image coordinates: where u r and v r represent the horizontal and vertical coordinates of the original right image, respectively.

[0145] Step S46: Interpolate the corrected image coordinates based on the original image coordinates to obtain the interpolated image coordinates.

[0146] Specifically, the interpolated image coordinates include the interpolated left camera image coordinates and the interpolated right camera image coordinates.

[0147] Further, the method for determining the interpolated left camera image coordinates is as follows:

[0148]

[0149] where are respectively rounding down and rounding up,

[0150] Further, the method for determining the interpolated right camera image coordinates is as follows:

[0151]

[0152] where are respectively rounding down and rounding up,

[0153] Step S47: Remove the coordinates with pixel values less than 0 in the interpolated image coordinates to obtain a cropped image; the aspect ratio of the cropped image is the same as that of the two frames of images captured by the binocular camera.

[0154] Specifically, the interpolated images of the left camera and the right camera are respectively cropped to remove the coordinates with pixel values less than 0, and the aspect ratio of the cropped images is

[0155] Step S48: Enlarge the size of the cropped image proportionally to the same size as the two frames of images captured by the binocular camera to obtain a vertically parallax-corrected image.

[0156] Specifically, the sizes of the cropped images of the left camera and the right camera should be kept the same, and then enlarged proportionally to W and H.

[0157] Step S5: Calculate the horizontal parallax statistical value according to the vertically parallax-corrected image.

[0158] Step S5 specifically includes:

[0159] Step S51: Calculate the horizontal parallax image according to the vertically parallax-corrected image.

[0160] Specifically, algorithms such as NCC, BM, and SGBM can be used. When performing stereo matching calculation on the left and right images, only horizontal traversal is required to find, which can save a lot of computing time.

[0161] Step S52: Perform weighted calculation of the parallax geometric position on the horizontal parallax image to obtain the horizontal parallax statistical value.

[0162] Specifically, the human eye pays more attention to the middle area of the video than the surrounding area. The parallax statistical value is obtained by using the method of geometric position weighting of parallax, and the weight where f is the Euclidean distance between the point (i, j) and the center point of the image The parallax statistical value is obtained by multiplying the weight by the parallax map, summing and normalizing.

[0163] Step S6: Adjust the horizontal parallax according to the horizontal parallax statistical value to obtain the adjusted 3D video image.

[0164] For monitors of different sizes and viewing distances, the 3D effect is also different. When a person observes a monitor, if the parallax between the left and right images on the monitor satisfies the fusional area range, the human eye observes more comfortably and is easy to focus. To meet the conditions, the parallax range of the left and right images must satisfy where η = 2.907×10 4 rad, is a constant, S is the distance between the observer and the monitor, D = 0.004(m) is the pupil diameter, Ee = 0.065(m) is the distance between the two pupils of the human eye, and P w is the width of a unit pixel of the monitor.

[0165] Step S6 specifically includes:

[0166] Step S61: When the left and right images of the vertically parallax-corrected image are respectively moved left and right by pixels to obtain the adjusted 3D video image.

[0167] Step S62: When the left and right images of the vertically parallax-corrected image are respectively moved right and left by to obtain the adjusted 3D video image.

[0168] Step S63: When the left and right images of the vertically parallax-corrected image are not adjusted at all to obtain the adjusted 3D video image.

[0169] where η = 2.907×10 4 rad, is a constant, S is the distance between the observer and the monitor, D = 0.004m is the pupil diameter of the observer, Ee = 0.065m is the distance between the two pupils of the observer, and P w is the width of a unit pixel of the monitor, Diff mean is the horizontal parallax statistical value, and s is the minimum parallax value. Further, s = 0.2*S.

[0170] In addition, the horizontal parallax between the left and right images is the difference in the abscissa of the pixels of the same natural object in the images captured by the right camera and the left camera. In this embodiment, the minimum parallax value is the minimum horizontal parallax statistical value.

[0171] As Figure 5 shown, the 3D camera system real-time video processing and stereoscopic sense adjustment system provided by the present invention includes:

[0172] A calibration module 1 for calibrating the binocular camera according to the checkerboard image by using the Zhang Dingyou calibration method to obtain the internal parameter matrix, distortion parameters and external parameter matrix; the binocular camera includes a left camera and a right camera.

[0173] A correction rotation matrix determination module 2 for obtaining the correction rotation matrix of the binocular camera according to the external parameter matrix.

[0174] A reprojection module 3 for performing stereoscopic reprojection on two frames of images captured by the binocular camera to obtain the projected image coordinates.

[0175] A vertical parallax correction image determination module 4 for obtaining the vertical parallax correction image according to the correction rotation matrix, internal parameter matrix, distortion parameters of the binocular camera and the projected image coordinates.

[0176] A horizontal parallax statistical value determination module 5 for calculating the horizontal parallax statistical value according to the vertical parallax correction image.

[0177] An adjustment module 6 for adjusting the horizontal parallax according to the horizontal parallax statistical value to obtain the adjusted 3D video image.

[0178] Among them, the correction rotation matrix determination module 2 includes:

[0179] A camera coordinate system rotation matrix and translation vector determination unit for calculating the camera coordinate system rotation matrix and translation vector of the left camera relative to the right camera according to the external parameter matrix.

[0180] A rotation matrix statistical value determination unit for calculating the mean value of the rotation matrix based on the geodesic distance according to the rotation matrix to obtain the rotation matrix statistical value.

[0181] A translation vector statistical value determination unit for calculating the Euler space mean value according to the translation vector to obtain the translation vector statistical value.

[0182] A correction rotation matrix determination unit of the binocular camera for obtaining the correction rotation matrix of the binocular camera by using the Bouguet algorithm according to the rotation matrix statistical value and the translation vector statistical value.

[0183] Among them, the vertical parallax correction image determination module 4 includes:

[0184] The binocular camera coordinate determination unit is used to multiply the projected image coordinates by the inverse matrix of the internal parameter matrix to obtain the binocular camera coordinates.

[0185] The binocular camera coordinate determination unit before stereo rectification is used to multiply the binocular camera coordinates by the inverse matrix of the rectification rotation matrix of the binocular camera to obtain the binocular camera coordinates before stereo rectification.

[0186] The normalization unit is used to normalize the binocular camera coordinates before stereo rectification to obtain the normalized coordinates.

[0187] The original binocular camera coordinate determination unit is used to add the normalized coordinates and the distortion parameters to obtain the original binocular camera coordinates.

[0188] The original image coordinate determination unit is used to multiply the original binocular camera coordinates by the internal parameter matrix to obtain the original image coordinates.

[0189] The interpolated image coordinate determination unit is used to interpolate the rectified image coordinates according to the original image coordinates to obtain the interpolated image coordinates.

[0190] The cropping unit is used to remove the coordinates with pixel values less than 0 in the interpolated image coordinates to obtain the cropped image; the aspect ratio of the cropped image is the same as the aspect ratio of the two frames of images captured by the binocular camera.

[0191] The vertical parallax correction image determination unit is used to scale up the size of the cropped image proportionally to the same size as the two frames of images captured by the binocular camera to obtain the vertical parallax correction image.

[0192] Among them, the horizontal parallax statistical value determination module 5 includes:

[0193] The horizontal parallax image determination unit is used to calculate the horizontal parallax image according to the vertical parallax correction image.

[0194] The horizontal parallax statistical value determination unit is used to perform weighted calculation of the parallax geometric position on the horizontal parallax image to obtain the horizontal parallax statistical value.

[0195] Among them, the adjustment module 6 includes:

[0196] The first adjustment unit is used to, when , move the left and right images of the vertical parallax correction image to the left and right respectively by pixels to obtain the adjusted 3D video image.

[0197] The second adjustment unit is used to, when , move the left and right images of the vertical parallax correction image to the right and left respectively by to obtain the adjusted 3D video image.

[0198] The third adjustment unit is used to, when When it is, the left and right images of the vertically parallax-corrected image are not adjusted, and the adjusted 3D video image is obtained.

[0199] Among them, η = 2.907×10 4 rad, which is a constant, S is the distance between the observer and the display, D is the pupil diameter, Ee is the distance between the two pupils of the human eye, P w is the width of a unit pixel of the display, Diff mean is the horizontal parallax statistical value, and s is the minimum parallax value.

[0200] Specifically, η = 2.907×10 4 rad, which is a constant, S is the distance between the observer and the display, D = 0.004m is the pupil diameter of the observer, Ee = 0.065m is the distance between the two pupils of the observer's eyes, P w is the width of a unit pixel of the display, Diff mean is the horizontal parallax statistical value, s is the minimum parallax value, and s = 0.2*S.

[0201] In addition, the horizontal parallax of the left and right images is the difference in the abscissas of the pixels where the same natural object is located in the image taken by the right camera and the image taken by the left camera. In this embodiment, the minimum parallax value is the minimum horizontal parallax statistical value.

[0202] The effects of the 3D camera system, real-time video processing, and stereoscopic effect adjustment method and system provided by the present invention are as follows:

[0203] 1. The method proposed by the present invention combines parallax adjustment, image distortion correction, and binocular stereo correction, and comprehensively considers the influencing factors of distortion, horizontal parallax, vertical parallax, display size, and viewing distance on viewing comfort.

[0204] 2. By correcting the image distortion, the image display becomes more realistic.

[0205] 3. Directly use the external parameters of multiple checkerboard images after distortion correction to calculate the binocular stereo correction rotation matrix, which is more convenient to eliminate vertical parallax, and at the same time greatly reduces the time-consuming of searching and matching the left and right image blocks in the process of calculating the parallax map. Then, perform real-time adjustment of the horizontal parallax to eliminate the dizziness caused by excessive horizontal parallax, and at the same time enlarge the case of smaller parallax to increase the stereoscopic effect.

[0206] 4. The horizontal parallax can be adaptively adjusted according to the display size and viewing distance, and the minimum viewing distance s is set as a parameter for the user to adjust, making it easier to achieve the purpose of human eye comfort.

[0207] In the present specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0208] In this article, specific examples are used to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation on the present invention.

Claims

1. A real-time video processing and stereoscopic sense adjustment method for a 3D camera system, characterized in that, The method includes: Calibrating the binocular camera using the Zhang Dingyou calibration method based on the checkerboard image to obtain the internal parameter matrix, distortion parameters, and external parameter matrix; the binocular camera includes a left camera and a right camera; Obtaining the corrected rotation matrix of the binocular camera according to the external parameter matrix; Performing stereo reprojection on two frames of images captured by the binocular camera to obtain the reprojected image coordinates, specifically including: The coordinates of each pixel after reprojection are found through reverse reprojection transformation to the original image coordinates and bilinear interpolation processing to obtain the final reprojected image and the reprojected image coordinates; Obtaining the vertical disparity correction image according to the corrected rotation matrix of the binocular camera, the internal parameter matrix, the distortion parameters, and the reprojected image coordinates, specifically including: Multiplying the reprojected image coordinates by the inverse matrix of the internal parameter matrix to obtain the binocular camera coordinates; Multiplying the binocular camera coordinates by the inverse matrix of the corrected rotation matrix of the binocular camera to obtain the binocular camera coordinates before stereo correction; Normalizing the binocular camera coordinates before stereo correction to obtain the normalized coordinates; Adding the normalized coordinates to the distortion parameters to obtain the original binocular camera coordinates; Multiplying the original binocular camera coordinates by the internal parameter matrix to obtain the original image coordinates; Interpolating the corrected image coordinates according to the original image coordinates to obtain the interpolated image coordinates; Removing the coordinates with pixel values less than 0 in the interpolated image coordinates to obtain the cropped image; the aspect ratio of the cropped image is the same as the aspect ratio of the two frames of images captured by the binocular camera; Scaling the size of the cropped image proportionally to the same size as the two frames of images captured by the binocular camera to obtain the vertical disparity correction image; Calculating the horizontal disparity statistical value according to the vertical disparity correction image, specifically including: Calculating the horizontal disparity image according to the vertical disparity correction image; Performing weighted calculation of the geometric position of the horizontal disparity on the horizontal disparity image to obtain the horizontal disparity statistical value; Adjusting the horizontal disparity according to the horizontal disparity statistical value to obtain the adjusted 3D video image.

2. The real-time video processing and stereoscopic sense adjustment method of the 3D camera system according to claim 1, characterized in that, The obtaining the corrected rotation matrix of the binocular camera according to the external parameter matrix specifically includes: Calculating the camera coordinate system rotation matrix and translation vector of the left camera relative to the right camera according to the external parameter matrix; Calculating the mean value of the rotation matrix based on the geodesic distance according to the rotation matrix to obtain the rotation matrix statistical value; Calculating the Euler space mean value according to the translation vector to obtain the translation vector statistical value; Obtaining the corrected rotation matrix of the binocular camera using the Bouguet algorithm according to the rotation matrix statistical value and the translation vector statistical value.

3. The real-time video processing and stereoscopic sense adjustment method of the 3D camera system according to claim 1, characterized in that, The adjusting the horizontal disparity according to the horizontal disparity statistical value to obtain the adjusted 3D video image specifically includes: When the left and right images of the vertical parallax correction image move left and right respectively by pixels to obtain an adjusted 3D video image; When the left and right images of the vertical parallax correction image are respectively moved to the right and left by to obtain an adjusted 3D video image; When the left and right images of the vertical parallax correction image are not adjusted at all to obtain an adjusted 3D video image; Among them, η = 2.907×10 4 rad, which is a constant, S is the distance between the observer and the display, D is the pupil diameter of the observer, Ee is the distance between the two pupils of the observer, P w is the width of a unit pixel of the display, Diff mean is the statistical value of the horizontal parallax, and s is the minimum parallax value.

4. A real-time video processing and stereoscopic sense adjustment system for a 3D camera system, characterized in that, The system includes: A calibration module for calibrating the binocular camera using the Zhang Dingyou calibration method based on the checkerboard image to obtain the internal parameter matrix, distortion parameters, and external parameter matrix; the binocular camera includes a left camera and a right camera; A corrected rotation matrix determination module for obtaining the corrected rotation matrix of the binocular camera according to the external parameter matrix; A reprojection module, which is used to perform stereo reprojection on two frames of images captured by the binocular camera to obtain the projected image coordinates; A vertical disparity correction image determination module, which is used to obtain a vertical disparity correction image according to the corrected rotation matrix of the binocular camera, the internal parameter matrix, the distortion parameters, and the projected image coordinates. Specifically, it includes: multiplying the projected image coordinates by the inverse matrix of the internal parameter matrix to obtain the binocular camera coordinates; Multiplying the binocular camera coordinates by the inverse matrix of the corrected rotation matrix of the binocular camera to obtain the binocular camera coordinates before stereo correction; Normalizing the binocular camera coordinates before stereo correction to obtain normalized coordinates; Adding the normalized coordinates to the distortion parameters to obtain the original binocular camera coordinates; Multiplying the original binocular camera coordinates by the internal parameter matrix to obtain the original image coordinates; Interpolating the corrected image coordinates according to the original image coordinates to obtain the interpolated image coordinates; Removing the coordinates with pixel values less than 0 in the interpolated image coordinates to obtain a cropped image; the aspect ratio of the cropped image is the same as the aspect ratio of the two frames of images captured by the binocular camera; Scaling the size of the cropped image proportionally to the same size as the two frames of images captured by the binocular camera to obtain the vertical disparity correction image; A horizontal disparity statistical value determination module, which is used to calculate the horizontal disparity statistical value according to the vertical disparity correction image. Specifically, it includes: calculating a horizontal disparity image according to the vertical disparity correction image; Performing a weighted calculation of the geometric position of the disparity on the horizontal disparity image to obtain the horizontal disparity statistical value; An adjustment module, which is used to adjust the horizontal disparity according to the horizontal disparity statistical value to obtain an adjusted 3D video image.

5. The real-time video processing and stereoscopic sense adjustment system of the 3D camera system according to claim 4, characterized in that, The corrected rotation matrix determination module includes: A camera coordinate system rotation matrix and translation vector determination unit, which is used to calculate the camera coordinate system rotation matrix and translation vector of the left camera relative to the right camera according to the external parameter matrix; A rotation matrix statistical value determination unit, which is used to calculate the mean value of the rotation matrix based on the geodesic distance according to the rotation matrix to obtain the rotation matrix statistical value; A translation vector statistical value determination unit, which is used to calculate the Euler space mean value according to the translation vector to obtain the translation vector statistical value; A corrected rotation matrix determination unit for the binocular camera, which is used to obtain the corrected rotation matrix of the binocular camera by using the Bouguet algorithm according to the rotation matrix statistical value and the translation vector statistical value.

6. The real-time video processing and stereoscopic sense adjustment system of the 3D camera system according to claim 5, characterized in that, The vertical disparity correction image determination module includes: A binocular camera coordinate determination unit, which is used to multiply the projected image coordinates by the inverse matrix of the internal parameter matrix to obtain the binocular camera coordinates; A binocular camera coordinate determination unit before stereo correction, which is used to multiply the binocular camera coordinates by the inverse matrix of the corrected rotation matrix of the binocular camera to obtain the binocular camera coordinates before stereo correction; A normalization unit, which is used to normalize the binocular camera coordinates before stereo correction to obtain normalized coordinates; An original binocular camera coordinate determination unit, which is used to add the normalized coordinates to the distortion parameters to obtain the original binocular camera coordinates; An original image coordinate determination unit, configured to multiply the original binocular camera coordinates by the internal parameter matrix to obtain original image coordinates; An interpolated image coordinate determination unit, configured to perform interpolation on the corrected image coordinates according to the original image coordinates to obtain interpolated image coordinates; A cropping unit, configured to remove coordinates with pixel values less than 0 in the interpolated image coordinates to obtain a cropped image; the aspect ratio of the cropped image is the same as the aspect ratio of the two frames of images captured by the binocular camera; A vertical parallax correction image determination unit, configured to proportionally enlarge the size of the cropped image to be the same as the size of the two frames of images captured by the binocular camera to obtain the vertical parallax correction image.

7. The real-time video processing and stereoscopic sense adjustment system of the 3D camera system according to claim 4, characterized in that, The horizontal parallax statistical value determination module includes: A horizontal parallax image determination unit, configured to calculate a horizontal parallax image according to the vertical parallax correction image; A horizontal parallax statistical value determination unit, configured to perform weighted calculation of the parallax geometric position on the horizontal parallax image to obtain the horizontal parallax statistical value.

8. The real-time video processing and stereoscopic sense adjustment system of the 3D camera system according to claim 4, wherein The adjustment module includes: The first adjustment unit is configured to, when the left and right images of the vertical parallax correction image are respectively moved leftward and rightward by pixels to obtain an adjusted 3D video image; A second adjustment unit, configured to, when the left and right images of the vertically parallax-corrected image are respectively moved rightward and leftward by to obtain an adjusted 3D video image; A third adjustment unit, configured to, when the left and right images of the vertical parallax correction image are not adjusted at all to obtain an adjusted 3D video image; where n = 2.907×10 4 rad, which is a constant, S is the distance between the observer and the display, D is the pupil diameter of the observer, Ee is the distance between the two pupils of the observer, P w is the width of a unit pixel of the display, Diff mean is the statistical value of the horizontal parallax, and s is the minimum parallax value.

Images