Method for improving generalization of stereo matching network based on optical axis offset

Abstract

The invention discloses a method for improving generalization of binocular cameras in mass production based on random optical axis offset, and the method comprises the steps: carrying out the random sampling of the binocular cameras, and carrying out the quantitative analysis of the optical axis offset angle of the batch of products after the binocular calibration; according to a statistical result of the camera deviation angle, calculating a mean value and a variance of the deviation angle, and constructing a rotation matrix meeting normal distribution through the mean value and the variance; remapping the right image according to the rotation matrix to obtain a left image and a right image which conform to real distribution and have optical axis offset; the model is trained on the data set which simulates the optical axis deviation, and the trained model is deployed in a binocular camera. According to the method, the angle of the optical axis deviation of the mass-production binocular camera in the actual working scene is counted, the data set is remapped, the data distribution when the optical axis deviation occurs in the real scene is simulated, the generalization of the model is improved, and the good reasoning effect can still be kept when the camera structure is slightly disturbed.

Description

Technical Field

[0001] The present invention relates to the field of binocular depth estimation, and in particular to a method for improving the generalization of a stereo matching network. Background Art

[0002] Binocular depth estimation is an important research direction in the field of computer vision, which aims to estimate the depth information of different points in the scene through images captured by two cameras (left and right cameras). Binocular depth estimation can be used in many applications, such as 3D reconstruction, stereo vision, autonomous driving, robot navigation, etc.

[0003] Early stereo depth estimation methods were mainly based on the principle of disparity calculation, such as block matching algorithms. These methods calculate disparity by comparing the differences between corresponding pixels in the left and right camera images, and then infer depth information. However, due to the limitations of the accuracy and robustness of disparity calculation, these methods have limited depth estimation effects in complex scenes.

[0004] In recent years, deep learning methods have made significant progress in the field of stereo depth estimation. Deep learning models can learn the mapping relationship between image features and depth through large-scale training data. These methods can learn depth information directly from images, avoiding the manually designed feature extraction process. Models such as convolutional neural networks (CNNs) and autoencoders have been widely used in depth estimation tasks and have achieved impressive performance.

[0005] However, there are still some challenges and shortcomings in binocular depth estimation. After searching, it was found that the Chinese patent application with publication number CN109741388A provides a method and device for generating a binocular depth estimation model. This method only provides a method for depth estimation under the condition that the left and right images meet the epipolar constraint, and does not consider the situation when the left and right images do not meet the epipolar constraint due to hardware reasons during the reasoning process of the camera, such as inaccurate installation or unstable camera fixed position. In practical applications, the reasoning effect of binocular depth estimation may be deteriorated due to the offset of the camera optical axis, which has a negative impact on the accuracy and robustness of binocular depth estimation.

[0006] In order to maintain good reasoning results when the left and right images do not meet the epipolar constraints and improve the generalization of the depth estimation network, a feasible method is to use a dataset with optical axis offset for model training during the training phase, use the adjusted left and right images as input, and the corresponding depth image as the target label, and train the deep learning model to learn the mapping relationship from the input image to the depth image. In this way, the network model can learn how to perform accurate depth estimation in the case of optical axis offset. Summary of the invention

[0007] In order to overcome the problem that the reasoning effect of the binocular depth estimation network is deteriorated due to the optical axis offset of the camera, the present invention provides a method for improving the generalization of the stereo matching network based on the optical axis offset. By statistically analyzing the angles of optical axis offset of mass-produced binocular cameras and changing the distribution of left and right images in the training stage to simulate the situation when the optical axis offset occurs, the network model is trained to learn the mapping relationship from the input image to the depth image, thereby improving the generalization of the binocular depth estimation network.

[0008] To achieve the above object, the present invention discloses a method for improving the generalization of mass-produced binocular cameras based on random optical axis offset, the steps comprising: Step S10: Randomly sample the binocular cameras and quantitatively analyze the angle of optical axis deviation of the batch of products after binocular positioning; Step S101: Select a specially made calibration plate, which is usually a flat plate-shaped object with a series of calibration points or calibration checkerboards of known sizes drawn on it. Place the calibration plate in the camera field of view and ensure that the calibration plate covers all corners and depth ranges of the camera field of view to obtain as many calibration points as possible; Step S102: Use a binocular camera to capture multiple images containing the calibration plate at different positions and angles. For each image, detect the calibration points on the calibration plate or the corner points of the calibration chessboard; Step S103: Using the image coordinates of the detected calibration points and the known world coordinates of the calibration points, the camera calibration algorithm is applied to calculate the internal and external parameters of the camera. By analyzing the rotation matrix, the camera's optical axis offset angle can be extracted, that is, the offset angle of the camera's optical axis relative to the world coordinate system, including the pitch angle , yaw angle , Roll Angle ; Step S20: Calculate the camera pitch angles according to the statistical results of step S10 , yaw angle , Roll Angle The mean and variance of , through which a rotation matrix satisfying normal distribution is constructed; Step S30: remapping the right image according to the rotation matrix to obtain left and right images that conform to the real distribution and have an optical axis offset; Step S301: reprojecting the right image from the pixel coordinate system to the camera coordinate system by left-multiplying the coordinate vector in the pixel coordinate system by the inverse matrix of the intrinsic parameter matrix; Step S302: multiplying the coordinate vector in the camera coordinate system by the rotation matrix to simulate the optical axis offset of the camera in a real scene; Step S303: projecting the right image from the camera coordinate system to the pixel coordinate system by left-multiplying the coordinate vector in the camera coordinate system by the intrinsic parameter matrix, thereby obtaining the left and right images after the optical axis offset occurs; Step S40: The model is trained on a data set that simulates an optical axis offset, and the trained model is deployed in a binocular camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 The flowchart of the steps for improving the generalization of mass-produced binocular cameras in the present invention.

[0010] Figure 2 It is a schematic diagram of simulating optical axis offset remapping according to the present invention. DETAILED DESCRIPTION

[0011] The following describes the implementation of the present invention through specific examples and in conjunction with the accompanying drawings. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0012] Figure 1 FIG. 1 is a flow chart of the steps of a method for improving the generalization of a stereo matching network based on optical axis offset according to the present invention. Figure 1 As shown, the specific steps of this method are: Step S10: Randomly sample the binocular cameras and quantitatively analyze the angle of optical axis deviation of the batch of products after binocular positioning; Step S101: Select a specially made calibration plate, which is usually a flat plate-shaped object with a series of calibration points or calibration checkerboards of known sizes drawn on it. Place the calibration plate in the camera field of view and ensure that the calibration plate covers all corners and depth ranges of the camera field of view to obtain as many calibration points as possible; Step S102: Use a binocular camera to capture multiple images containing the calibration plate at different positions and angles. For each image, detect the calibration points on the calibration plate or the corner points of the calibration chessboard; Step S103: Using the image coordinates of the detected calibration points and the known world coordinates of the calibration points, refer to the Zhang Zhengyou calibration method to calculate the internal and external parameters of the camera. By analyzing the rotation matrix, the camera's optical axis offset angle can be extracted, that is, the offset angle of the camera's optical axis relative to the world coordinate system, including the pitch angle , yaw angle , Roll Angle ; Step S20: Calculate the mean of the camera pitch angle, yaw angle, and roll angle according to the statistical results of step S10 , , With variance , , , construct the rotation matrix R that satisfies the normal distribution through the mean and variance;

[0013] in, , , Represents the camera pitch angle , yaw angle , Roll Angle The sub-rotation matrix of The value is Random values ​​in The value is Random values ​​in The value is Random values ​​in ; Step S30: remapping the right image according to the rotation matrix to obtain left and right images that conform to the real distribution and have an optical axis offset; Step S301: reprojecting the right image from the pixel coordinate system to the camera coordinate system by left-multiplying the coordinate vector in the pixel coordinate system by the inverse matrix K of the intrinsic parameter matrix;

[0014] in, and is the focal length of the camera, and is the coordinate of the optical center of the camera, Represents the horizontal and vertical coordinates of the pixel coordinate system in the right image. Represents the horizontal, vertical and vertical coordinates of the camera coordinate system in the right image; Step S302: multiplying the coordinate vector in the camera coordinate system by the rotation matrix to simulate the optical axis offset of the camera in a real scene;

[0015] in, Represents the horizontal, vertical and vertical coordinates after the optical axis is offset in the camera coordinate system in the right figure; Step S303: projecting the right image from the camera coordinate system to the pixel coordinate system by left-multiplying the coordinate vector in the camera coordinate system by the intrinsic parameter matrix, thereby obtaining the left and right images after the optical axis offset occurs;

[0016] in, The horizontal and vertical coordinates after the optical axis is offset in the pixel coordinate system in the right figure; Step S40: referring to the common stereo matching network training method in the field, the model is trained on a data set simulating an optical axis offset, and the trained model is deployed in the binocular camera.

Claims

1. A method for improving the generalization of a stereo matching network for mass-produced binocular cameras based on random optical axis offset, characterized in that: By quantitatively analyzing the optical axis offset angle of mass-produced binocular cameras after binocular localization, an incompletely registered data set is constructed for training to improve the generalization of the stereo matching network, including the following steps: S10: Ideally, the left and right camera image planes of a stereo-calibrated binocular camera are parallel and meet the coplanar alignment. However, due to inaccurate camera installation or unstable camera fixed position, optical axis offset may occur, resulting in the corresponding points of the left and right images not being in the same row. Random sampling of binocular cameras is performed to quantitatively analyze the optical axis offset angle of this batch of products after binocular alignment. S20: Calculate the mean and variance of the offset angles according to the statistical results of the camera offset angles in step S10, and construct a rotation matrix that satisfies the normal distribution through the mean and variance; S30: remapping the right image according to the rotation matrix to obtain left and right images that conform to the real distribution and have an optical axis offset; S40: The model is trained on a dataset that simulates optical axis offset, and the trained model is deployed in the binocular camera.

2. The method for simulating optical axis offset remapping according to claim 1, characterized in that: The right image is reprojected from the pixel coordinate system to the camera coordinate system by multiplying the coordinate vector in the pixel coordinate system by the inverse matrix of the intrinsic parameter matrix on the left; the coordinate vector in the camera coordinate system is multiplied by the rotation matrix on the left to simulate the optical axis offset of the camera in the real scene; the right image is projected from the camera coordinate system to the pixel coordinate system by multiplying the coordinate vector in the camera coordinate system by the intrinsic parameter matrix on the left to obtain the left and right images after the optical axis offset.

Images