Binocular vision sensor calibration method, device, computer equipment and storage medium

Abstract

The present application relates to a binocular vision sensor calibration method, device, computer equipment and storage medium. The method comprises: obtaining each cross laser line image group obtained by the binocular vision sensor to be calibrated under different shooting conditions, including a left cross laser line image and a right cross laser line image; determining the left image intersection point of the cross laser line in the left cross laser line image, and determining the right image intersection point of the cross laser line in the right cross laser line image; obtaining the feature point matching pair corresponding to the cross laser line image group according to the left image intersection point and the right image intersection point; determining the position mapping matrix between the left cross laser line image and the right cross laser line image according to each feature point matching pair and the internal parameters of the binocular vision sensor; decomposing the position mapping matrix to obtain the external parameters of the binocular vision sensor, and the calibration result of the binocular vision sensor includes the external parameters. The use of this method can improve the calibration efficiency.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a binocular vision sensor calibration method, device, computer equipment and storage medium. Background Art

[0002] Binocular Stereo Vision is an important form of machine vision. It is based on the principle of parallax and uses binocular vision sensors to obtain two images of the object being measured from different positions. By calculating the position deviation between corresponding points in the images, the three-dimensional geometric information of the object is obtained. The binocular stereo vision measurement method has the advantages of high efficiency, appropriate accuracy, simple system structure, and low cost. It is very suitable for online, non-contact product inspection and quality control at the manufacturing site.

[0003] In order to obtain accurate binocular stereo vision measurement results, it is usually necessary to calibrate and calibrate the positional relationship between binocular vision sensors. The traditional binocular calibration method based on planar targets requires the placement of numerous calibration plates within the field of view of the binocular vision sensor, which is labor-intensive and has low calibration efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a binocular vision sensor calibration method, device, computer equipment and storage medium that can improve calibration efficiency in response to the above technical problems.

[0005] A binocular vision sensor calibration method, the method comprising:

[0006] Acquire each cross laser line image group respectively captured by the binocular vision sensor to be calibrated under different shooting conditions, the cross laser line image group including a left cross laser line image and a right cross laser line image;

[0007] Determine a left image intersection point of the cross laser line in the left cross laser line image, and determine a right image intersection point of the cross laser line in the right cross laser line image;

[0008] According to the intersection points of the left image and the right image, a matching pair of feature points corresponding to the cross laser line image group is obtained;

[0009] Determine the position mapping matrix between the left cross laser line image and the right cross laser line image according to the matching pairs of feature points and the internal parameters of the binocular vision sensor;

[0010] The position mapping matrix is ​​decomposed to obtain the external parameters of the binocular vision sensor. The calibration result of the binocular vision sensor includes the external parameters.

[0011] In one embodiment, obtaining each cross laser line image group captured by the binocular vision sensor to be calibrated under different shooting conditions includes:

[0012] When the binocular vision sensor to be calibrated is at different sensor distribution parameters, the position of the preset cross laser is adjusted to obtain each cross laser line image group respectively photographed by the binocular vision sensor.

[0013] In one embodiment, determining a left image intersection point of a cross laser line in a left cross laser line image and determining a right image intersection point of a cross laser line in a right cross laser line image comprises:

[0014] The left cross laser line image and the right cross laser line image are filtered to obtain a left denoised image and a right denoised image respectively;

[0015] Laser straight line fitting is performed on the cross laser lines in the left denoised image and the right denoised image, and the intersection points of the left image and the right image are determined respectively according to the straight line fitting results.

[0016] In one embodiment, the intrinsic parameters of the binocular vision sensor include an internal parameter matrix; the position mapping matrix includes a homography matrix; and determining the position mapping matrix between the left cross laser line image and the right cross laser line image according to each feature point matching pair and the intrinsic parameters of the binocular vision sensor includes:

[0017] Determine a sensor coordinate mapping function between feature points in each feature point matching pair;

[0018] According to the image coordinates of the feature points and the internal parameter matrix, the sensor coordinates corresponding to the feature points are determined, and the sensor coordinates are the coordinates of the feature points in the corresponding sensor coordinate system;

[0019] According to the sensor coordinate mapping function and the sensor coordinates, the homography matrix between the left cross laser line image and the right cross laser line image is obtained.

[0020] In one embodiment, before determining the position mapping matrix between the left cross laser line image and the right cross laser line image according to the matching pairs of feature points and the internal parameters of the binocular vision sensor, the method further includes:

[0021] The binocular vision sensor is calibrated internally to obtain the internal parameters of the binocular vision sensor.

[0022] In one embodiment, decomposing the position mapping matrix to obtain the external parameters of the binocular vision sensor includes:

[0023] Decompose the homography matrix by singular value decomposition algorithm to obtain the matrix decomposition result;

[0024] The positional relationship of the matrix decomposition results is logically screened, and the external parameters of the binocular vision sensor are obtained according to the logical screening results.

[0025] In one embodiment, the intrinsic parameters of the binocular vision sensor also include a radial distortion factor; after decomposing the position mapping matrix to obtain the extrinsic parameters of the binocular vision sensor, it also includes:

[0026] Determine the cost function of the external parameters through the internal parameter matrix, radial distortion factor, sensor distribution parameters of the binocular vision sensor and image coordinates of each feature point matching pair;

[0027] The external parameters are adjusted according to the cost function to obtain the optimized external parameters, and the optimized external parameters are used as the calibration results of the binocular vision sensor.

[0028] A binocular vision sensor calibration device, the device comprising:

[0029] An image group acquisition module is used to acquire each cross laser line image group respectively captured by the binocular vision sensor to be calibrated under different shooting conditions, wherein the cross laser line image group includes a left cross laser line image and a right cross laser line image;

[0030] An intersection point determination module is used to determine the intersection point of the left image of the cross laser line in the left cross laser line image, and determine the intersection point of the right image of the cross laser line in the right cross laser line image;

[0031] A matching pair determination module, used to obtain a matching pair of feature points corresponding to the cross laser line image group according to the intersection points of the left image and the intersection points of the right image;

[0032] A mapping matrix determination module is used to determine the position mapping matrix between the left cross laser line image and the right cross laser line image according to each feature point matching pair and the internal parameters of the binocular vision sensor;

[0033] The calibration result acquisition module is used to decompose the position mapping matrix to obtain the external parameters of the binocular vision sensor. The calibration result of the binocular vision sensor includes the external parameters.

[0034] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Acquire each cross laser line image group respectively captured by the binocular vision sensor to be calibrated under different shooting conditions, the cross laser line image group including a left cross laser line image and a right cross laser line image;

[0036] Determine a left image intersection point of the cross laser line in the left cross laser line image, and determine a right image intersection point of the cross laser line in the right cross laser line image;

[0037] According to the intersection points of the left image and the right image, a matching pair of feature points corresponding to the cross laser line image group is obtained;

[0038] Determine the position mapping matrix between the left cross laser line image and the right cross laser line image according to the matching pairs of feature points and the internal parameters of the binocular vision sensor;

[0039] The position mapping matrix is ​​decomposed to obtain the external parameters of the binocular vision sensor. The calibration result of the binocular vision sensor includes the external parameters.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0041] Acquire each cross laser line image group respectively captured by the binocular vision sensor to be calibrated under different shooting conditions, the cross laser line image group including a left cross laser line image and a right cross laser line image;

[0042] Determine a left image intersection point of the cross laser line in the left cross laser line image, and determine a right image intersection point of the cross laser line in the right cross laser line image;

[0043] According to the intersection points of the left image and the right image, a matching pair of feature points corresponding to the cross laser line image group is obtained;

[0044] Determine the position mapping matrix between the left cross laser line image and the right cross laser line image according to the matching pairs of feature points and the internal parameters of the binocular vision sensor;

[0045] The position mapping matrix is ​​decomposed to obtain the external parameters of the binocular vision sensor. The calibration result of the binocular vision sensor includes the external parameters.

[0046] The above binocular vision sensor calibration method, device, computer equipment and storage medium are based on each cross laser line image group obtained by the binocular vision sensor to be calibrated under different shooting conditions, determine the corresponding feature point matching pair, and determine the position mapping matrix between the left cross laser line image and the right cross laser line image with the feature point matching pair, decompose the position mapping matrix, obtain the external parameters of the binocular vision sensor, and realize the calibration of the binocular vision sensor. The corresponding feature point matching pairs can be quickly and accurately determined by each cross laser line image group obtained by shooting under different shooting conditions, and then the position mapping matrix is ​​determined based on the feature point matching pair and the external parameters are further obtained, so as to realize the rapid calibration of the binocular vision sensor and improve the calibration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A diagram showing an application environment of a binocular vision sensor calibration method in one embodiment;

[0048] Figure 2 A schematic diagram of a process flow of a binocular vision sensor calibration method in one embodiment;

[0049] Figure 3 A schematic diagram of a process of determining a position mapping matrix in one embodiment;

[0050] Figure 4 A schematic diagram of a decomposition result of performing SVD decomposition processing on a homography matrix in one embodiment;

[0051] Figure 5 It is a flowchart of a binocular vision sensor calibration method in another embodiment;

[0052] Figure 6 is a structural block diagram of a binocular vision sensor calibration device in one embodiment;

[0053] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] The binocular vision sensor calibration method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the binocular vision sensor to be calibrated is composed of a left camera A and a right camera B, C is a cross laser, and the cross laser emits a cross laser line to a plane. The left camera A and the right camera B of the binocular vision sensor shoot the cross laser line on the plane, and adjust different shooting conditions, and send each cross laser line image group obtained by shooting to the terminal (not shown) or the server (not shown) for calibration processing. The terminal or server receives each cross laser line image group including the left cross laser line image and the right cross laser line image, determines the corresponding feature point matching pair, and determines the position mapping matrix between the left cross laser line image and the right cross laser line image according to the feature point matching pair, decomposes the position mapping matrix, and obtains the external parameters of the binocular vision sensor, such as Figure 1 The rotation matrix R and translation vector t shown are used to calibrate the binocular vision sensor.

[0056] The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0057] In one embodiment, Figure 2 As shown, a binocular vision sensor calibration method is provided, comprising the following steps:

[0058] Step S201: acquiring each cross laser line image group captured by the binocular vision sensor to be calibrated under different shooting conditions, wherein the cross laser line image group includes a left cross laser line image and a right cross laser line image.

[0059] The binocular vision sensor consists of a left vision sensor and a right vision sensor. The vision sensor can be a camera, which is used to obtain two images of the object under test from different left and right positions, so as to obtain the three-dimensional geometric information of the object by calculating the position deviation between the corresponding points of the image, thereby realizing binocular stereo vision. Calibration is the process of determining the geometric model of the left vision sensor and the right vision sensor in the binocular vision sensor, that is, the camera parameters. Generally, for a binocular vision sensor, the external parameters between the image taken by the left vision sensor and the image taken by the right vision sensor need to be calibrated, which can specifically include a rotation matrix and a translation vector.

[0060] The binocular vision sensor to be calibrated shoots the cross laser line emitted by the preset cross laser under different shooting conditions, and sends the captured image to the terminal or server to obtain a cross laser line image group. Specifically, the preset cross laser can emit a cross laser line to a white plane (such as a painting wall, a billboard or a white flat wall, etc.), and the binocular vision sensor to be calibrated shoots the cross laser line on the white plane. For different shooting conditions, the position and angle between the left visual sensor and the right visual sensor in the binocular vision sensor can be adjusted, or the position of the cross laser can be adjusted to simulate different shooting conditions. For different shooting conditions, the binocular vision sensor shoots the cross laser line to obtain a cross laser line image group consisting of a left cross laser line image and a right cross laser line image, wherein the left cross laser line image is obtained by shooting the left visual sensor, and the right cross laser line image is obtained by shooting the right visual sensor.

[0061] Step S203: determining a left image intersection point of the cross laser line in the left cross laser line image, and determining a right image intersection point of the cross laser line in the right cross laser line image.

[0062] After obtaining each cross laser line image group including the left cross laser line image and the right cross laser line image, determine the left image intersection point of the cross laser line in the left cross laser line image, and determine the right image intersection point of the cross laser line in the right cross laser line image. Among them, the left image intersection point and the right image intersection point are the cross intersection points of the cross laser lines. Specifically, image processing can be performed on the left cross laser line image and the right cross laser line image, such as filtering denoising, straight line fitting, etc., to determine the left image intersection point and the right image intersection point respectively.

[0063] Step S205: obtaining a feature point matching pair corresponding to the cross laser line image group according to the intersection points of the left image and the intersection points of the right image.

[0064] In this embodiment, the cross intersection of the cross laser line is used as a feature point during calibration, and a feature point matching pair is constructed based on the feature point, so as to perform binocular vision sensor calibration based on the feature point matching pair. Specifically, the intersection point of the left image and the intersection point of the right image obtained under the same shooting condition are combined as the feature point matching pair corresponding to the cross laser line image group.

[0065] Step S207: determining a position mapping matrix between the left cross laser line image and the right cross laser line image according to each feature point matching pair and the internal parameters of the binocular vision sensor.

[0066] After obtaining the matching pairs of feature points, the position mapping matrix between the left cross laser line image and the right cross laser line image is determined respectively in combination with the internal parameters of the binocular vision sensor, such as the internal parameter matrix. The position mapping matrix reflects the relative position relationship between the left and right views in the binocular vision sensor under the assumption of the pinhole camera model. In specific implementation, the position mapping matrix can be a 3x3 homography matrix. When a planar object is simultaneously photographed by the left and right cameras of the binocular vision sensor, the homography matrix can be calculated and derived.

[0067] Step S209: decomposing the position mapping matrix to obtain the extrinsic parameters of the binocular vision sensor. The calibration result of the binocular vision sensor includes the extrinsic parameters.

[0068] After obtaining the position mapping matrix, the position mapping matrix is ​​decomposed to obtain the external parameters of the binocular vision sensor. The external parameters can be used as the calibration results of the binocular vision sensor, such as the rotation matrix and the translation vector. Specifically, when the position mapping matrix is ​​a 3x3 homography matrix, the homography matrix can be decomposed based on the SVD (Singular Value Decomposition) algorithm to obtain the rotation matrix and translation vector that meet the conditions, such as obtaining the correct solution that the left visual sensor and the right visual sensor are located on the same side of the singular value decomposition baseline and the translation transformation vectors of the two are opposite numbers, that is, the correct rotation matrix and translation vector. By projecting a cross laser line on a plane and extracting the intersection of the projected cross laser line, the feature point matching pair is obtained, which is efficient and not prone to mismatching, and has stronger robustness. The binocular vision sensor is used to shoot the cross laser line multiple times to obtain a large number of feature point matching pairs on the same plane. These matching point pairs are used to calculate the homography mapping matrix from the left visual sensor to the right visual sensor image plane. By decomposing the matrix, the external parameters of the binocular vision sensor can be obtained, thereby realizing the on-site rapid calibration of the binocular vision sensor.

[0069] In the above binocular vision sensor calibration method, based on each cross laser line image group obtained by the binocular vision sensor to be calibrated under different shooting conditions, the corresponding feature point matching pairs are determined, and the position mapping matrix between the left cross laser line image and the right cross laser line image is determined with the feature point matching pairs, and the position mapping matrix is ​​decomposed to obtain the external parameters of the binocular vision sensor, thereby realizing the calibration of the binocular vision sensor. The corresponding feature point matching pairs can be quickly and accurately determined by each cross laser line image group obtained by shooting under different shooting conditions, and then the position mapping matrix is ​​determined based on the feature point matching pairs and the external parameters are further obtained, thereby realizing the rapid calibration of the binocular vision sensor and improving the calibration efficiency.

[0070] In one of the embodiments, obtaining each cross laser line image group respectively photographed by the binocular vision sensor to be calibrated under different shooting conditions includes: when the binocular vision sensor to be calibrated is in different sensor distribution parameters, adjusting the position of a preset cross laser, and obtaining each cross laser line image group respectively photographed by the binocular vision sensor.

[0071] In this embodiment, by adjusting the sensor distribution parameters of the binocular vision sensor and the position of the preset cross laser, different shooting conditions are constructed to obtain various cross laser line image groups. Among them, the sensor distribution parameters may include the position and angle of the binocular vision sensor. Specifically, when the binocular vision sensor to be calibrated is in different sensor distribution parameters, the position of the preset cross laser is adjusted, and the cross laser is emitted from the cross laser at different positions, and the binocular vision sensor is used to shoot, and each cross laser line image group is obtained.

[0072] In specific implementation, a preset cross laser can be used to project a cross laser line onto a plane, and the left and right vision sensors of the binocular vision sensor can shoot the cross laser line on the plane, and then the cross laser can be slowly moved and the binocular vision sensor can be continuously and synchronously triggered to shoot, such as N1 shooting can be performed; then the angle and position of the left and right vision sensors of the binocular vision sensor can be adjusted, and the cross laser can be slowly moved again and the binocular vision sensor can be continuously and synchronously triggered to shoot N2 images, and the process can be repeated, so that the binocular vision sensor can finally adjust M angle positions in total. Among them, when moving the cross laser, ensure that the cross intersection of the cross laser line covers the overlapping area of ​​the field of view of the left and right vision sensors at the current binocular vision sensor position, so as to ensure that both the left and right vision sensors can shoot the cross laser line on the plane.

[0073] In one embodiment, determining the left image intersection point of the cross laser lines in the left cross laser line image and determining the right image intersection point of the cross laser lines in the right cross laser line image include: filtering the left cross laser line image and the right cross laser line image respectively to obtain a left denoised image and a right denoised image; performing laser straight line fitting on the cross laser lines in the left denoised image and the right denoised image, and determining the left image intersection point and the right image intersection point respectively according to the straight line fitting results.

[0074] In this embodiment, the left cross laser line image and the right cross laser line image are image processed respectively to determine the cross intersection points of the cross laser lines in the left cross laser line image and the right cross laser line image. Specifically, the left cross laser line image and the right cross laser line image are filtered respectively, such as using a median filtering algorithm to filter, remove image noise, and obtain a left denoised image and a right denoised image. Then, laser straight line fitting is performed on the cross laser lines in the left denoised image and the right denoised image, such as using Hough Transform or RANSAC algorithm (RANdom SAmple Consensus) or least squares method to perform laser straight line fitting on the cross laser lines, and the left image intersection point and the right image intersection point are determined respectively according to the straight line fitting results. In specific implementation, the left cross laser line image and the right cross laser line image are image processed respectively, and the image coordinates of the left image intersection point in the image coordinate system of the left cross laser line image and the image coordinates of the right image intersection point in the image coordinate system of the right cross laser line image can be determined. The intersection points of the left image and the intersection points of the right image under the same shooting conditions can form a feature point matching pair.

[0075] In one embodiment, if Figure 3 As shown, the step of determining the position mapping matrix, that is, determining the position mapping matrix between the left cross laser line image and the right cross laser line image according to each feature point matching pair and the internal parameters of the binocular vision sensor, includes:

[0076] Step S301: Determine a sensor coordinate mapping function between feature points in each feature point matching pair.

[0077] In this embodiment, the internal parameters of the binocular vision sensor include an internal parameter matrix, the position mapping matrix includes a homography matrix, and when determining the position mapping matrix, the sensor coordinate mapping function between the feature points in each feature point matching pair is determined, and the sensor coordinate mapping function can specifically be a mapping function between sensor coordinate systems, where the sensor coordinate system is a coordinate system constructed based on the vision sensor. For example, for a binocular vision sensor composed of left and right cameras, the left and right camera coordinate systems can be constructed respectively with the left and right cameras, that is, the sensor coordinate system.

[0078] In specific implementation, if the binocular vision sensor is composed of left and right cameras, assume that a point P on a plane in space has coordinates Q in the left camera coordinate system. L , the coordinate of the right camera coordinate system is Q R , then we have:

[0079] Q R =RQ L +t (1)

[0080] Among them, R is the rotation matrix from the left camera to the right camera, and t is the translation vector from the left camera to the right camera.

[0081] Assume that the normal vector of the plane in the left camera coordinate system is n. According to the point-to-surface distance formula, the distance between the plane and the origin of the coordinate system is d = n. T O L ,Right now:

[0082]

[0083] Substituting equation (2) into equation (1), we can get

[0084]

[0085] Formula (3) reflects the sensor coordinate mapping relationship between the feature points in each feature point matching pair, that is, the mapping relationship between the sensor coordinate systems.

[0086] Step S303: Determine the sensor coordinates corresponding to the feature points according to the image coordinates of the feature points and the internal parameter matrix, where the sensor coordinates are the coordinates of the feature points in the corresponding sensor coordinate system.

[0087] The image coordinates of the feature points are the coordinates of the feature points in the image coordinate system of the images captured by the respective visual sensors. For example, for a binocular visual sensor, the left and right image coordinate systems can be constructed based on the images captured by the left and right cameras. The image coordinates of the feature points are the coordinates of the feature points in the corresponding image coordinate system. The sensor coordinates are the coordinates of the feature points in the corresponding sensor coordinate system. The sensor coordinate system is the coordinate system constructed based on the visual sensor.

[0088] Specifically, if the binocular vision sensor is composed of left and right cameras, the internal parameter matrix includes the left camera internal parameter K L and the right camera intrinsic parameter K R , let Q L and Q R The coordinates in the left and right camera image coordinate systems are q L and q R ,but

[0089]

[0090] Formula (4) can be used to determine the sensor coordinates corresponding to the feature points based on the image coordinates of the feature points and the internal parameter matrix.

[0091] Step S305: Obtaining a homography matrix between the left cross laser line image and the right cross laser line image according to the sensor coordinate mapping function and the sensor coordinates.

[0092] The obtained sensor coordinate mapping function and the sensor coordinates are combined to obtain a homography matrix between the left cross laser line image and the right cross laser line image.

[0093] Specifically, as in the above example, when the binocular vision sensor consists of left and right cameras, substitute equation (4) into equation (3) to obtain

[0094]

[0095] make but

[0096] q R =Hq L (6)

[0097] Among them, there are 5 unknown quantities in the homography matrix H. Through 5 pairs of feature point matching pairs to form 10 sets of equations, the H matrix (3x3) result can be solved.

[0098] In one of the embodiments, before determining the position mapping matrix between the left cross laser line image and the right cross laser line image based on the matching pairs of feature points and the intrinsic parameters of the binocular vision sensor, it also includes: performing an internal parameter calibration process on the binocular vision sensor to obtain the intrinsic parameters of the binocular vision sensor.

[0099] When determining the sensor coordinates corresponding to the feature points according to the image coordinates of the feature points, the internal parameter matrix of the binocular vision sensor is required. In this embodiment, before determining the position mapping matrix, a step of determining the internal parameters of the binocular vision sensor is also included.

[0100] Specifically, the binocular vision sensor is calibrated with internal parameters, such as using Zhang Zhengyou's plane target calibration method to calibrate the left visual sensor and the right visual sensor in the binocular vision sensor respectively to obtain the internal parameters of the binocular vision sensor. The internal parameters may include the internal parameter matrix K L , K R and the radial distortion factor λ of the lens L , R .

[0101] In one of the embodiments, decomposing the position mapping matrix to obtain the external parameters of the binocular vision sensor includes: decomposing the homography matrix through a singular value decomposition algorithm to obtain a matrix decomposition result; performing logical screening on the positional relationship of the matrix decomposition result, and obtaining the external parameters of the binocular vision sensor based on the logical screening result.

[0102] In this embodiment, the homography matrix is ​​decomposed by a singular value decomposition algorithm to obtain a matrix decomposition result, and the positional relationship of the matrix decomposition result is logically screened, and then the external parameters of the binocular vision sensor are obtained according to the logical screening result.

[0103] When implementing it specifically, Figure 4 As shown in the figure, after SVD decomposition of the homography matrix H, 4 groups of R, t / d, and n can be obtained, corresponding to different relative position relationships. Considering that the left and right cameras need to be on the same side of the baseline, the above group (1) or (2) is a possible solution, and the translation transformation vectors t / d of the two are opposite to each other. Since the left camera must be to the left of the right camera, the first element of t / d is less than 0, and the remaining erroneous solutions can be excluded, thereby obtaining the accurate external parameters of the binocular vision sensor, namely the rotation matrix R and the translation vector t.

[0104] In one of the embodiments, the intrinsic parameters of the binocular vision sensor also include a radial distortion factor; after decomposing the position mapping matrix to obtain the extrinsic parameters of the binocular vision sensor, it also includes: determining a cost function of the extrinsic parameters through the internal parameter matrix, the radial distortion factor, the sensor distribution parameters of the binocular vision sensor and the image coordinates of each feature point matching pair; adjusting the external parameters according to the cost function to obtain optimized external parameters, and using the optimized external parameters as the calibration result of the binocular vision sensor.

[0105] In this embodiment, after obtaining the external parameters of the binocular vision sensor, the external parameters can also be nonlinearly optimized to improve the accuracy of the binocular vision sensor calibration result. Specifically, the internal parameters of the binocular vision sensor also include radial distortion factors. The cost function of the external parameters is determined by the internal parameter matrix, the radial distortion factor, the sensor distribution parameters of the binocular vision sensor, and the image coordinates of each feature point matching pair, and the obtained external parameters are optimized and adjusted according to the cost function. Among them, the cost function can be determined according to the cumulative projection error of the feature point matching pairs corresponding to each cross laser line image group.

[0106] In specific applications, in the above example, when the binocular vision sensor consists of left and right cameras, the cost function can be expressed by formula (7):

[0107]

[0108] Among them, p is the moving position number of the binocular vision sensor, N p is the number of images taken at the pth position, i is the sequence number of the images taken at each binocular vision sensor position, x i and x′ i is the image coordinate of each feature point matching pair. G is the projection transformation model, which transforms the point x on the left camera image coordinate system i The corresponding point x′ projected onto the right camera image coordinate system i , which can be expressed as follows:

[0109] G=g(K L, K R ,λ L ,λ R , R, t, d p , n p ) (8)

[0110] Among them, K L , K R are the left camera intrinsic parameters and the right camera intrinsic parameters respectively, λ L ,λ R is the radial distortion factor of the left and right cameras. R is the three-dimensional vector representing the rotation transformation from the left camera to the right camera using Euler angles, t is the three-dimensional vector representing the translation transformation, and d p is the distance from the left camera to the plane where the cross projection line is located at the pth position, n p It is the unit normal vector of the plane where the cross projection line is located in the left camera coordinate system at the pth position. R, t, d p , n p The homography transformation matrix from the left camera image coordinate system to the right camera image coordinate system is constructed while removing the influence of image distortion.

[0111] The internal parameter K during the optimization process L , K R , L , R It is known that R, t is fixed at any p positions, d p and n p It is different at different binocular vision sensor positions. Taking {R, t} as the parameter object to be optimized, a nonlinear optimization algorithm such as the Levenberg-Marquardt method or Lie algebra is used for nonlinear optimization to finally obtain the optimized R and t.

[0112] In one embodiment, Figure 5 As shown, a binocular vision sensor calibration method is provided, comprising:

[0113] Step S501: when the binocular vision sensor to be calibrated is at different sensor distribution parameters, adjusting the position of the preset cross laser, and acquiring each cross laser line image group respectively photographed by the binocular vision sensor;

[0114] Step S502: filtering the left cross laser line image and the right cross laser line image respectively to obtain a left denoised image and a right denoised image;

[0115] Step S503: performing laser straight line fitting on the cross laser lines in the left denoised image and the right denoised image, and determining the intersection point of the left image and the intersection point of the right image respectively according to the straight line fitting results;

[0116] Step S504: obtaining a feature point matching pair corresponding to the cross laser line image group according to the intersection points of the left image and the intersection points of the right image.

[0117] In this embodiment, a preset cross laser is projected onto a plane, and a plurality of different shooting conditions are constructed by adjusting the sensor distribution parameters of the binocular vision sensor and the position of the cross laser. The binocular vision sensor shoots the cross laser lines on the plane to obtain each cross laser line image group. The cross laser line image group includes a left cross laser line image and a right cross laser line image. The left cross laser line image and the right cross laser line image are filtered and processed respectively by using a median filtering algorithm to remove image noise, thereby obtaining a left denoised image and a right denoised image. Then, laser straight line fitting is performed on the cross laser lines in the left denoised image and the right denoised image by using a Hough transform, a RANSAC algorithm or a least squares method, and the intersection point of the left image and the intersection point of the right image are determined respectively according to the straight line fitting results. The intersection point of the left image and the intersection point of the right image under the same shooting conditions can form a feature point matching pair.

[0118] Step S505: perform internal parameter calibration processing on the binocular vision sensor to obtain the internal parameters of the binocular vision sensor.

[0119] The left and right vision sensors in the binocular vision sensor are calibrated respectively using Zhang Zhengyou's plane target calibration method to obtain the internal parameters of the binocular vision sensor. The internal parameters include the internal parameter matrix K L , K R and the radial distortion factor λ of the lens L , R .

[0120] Step S506: determining a position mapping matrix between the left cross laser line image and the right cross laser line image according to each feature point matching pair and the internal parameters of the binocular vision sensor;

[0121] Step S507: decomposing the homography matrix by a singular value decomposition algorithm to obtain a matrix decomposition result;

[0122] Step S508: performing logical screening on the positional relationship of the matrix decomposition result, and obtaining the external parameters of the binocular vision sensor according to the logical screening result.

[0123] In this embodiment, determining the position mapping matrix includes: determining the sensor coordinate mapping function between the feature points in each feature point matching pair; determining the sensor coordinates corresponding to the feature points according to the image coordinates of the feature points and the internal parameter matrix, the sensor coordinates being the coordinates of the feature points in the corresponding sensor coordinate system; obtaining the homography matrix between the left cross laser line image and the right cross laser line image according to the sensor coordinate mapping function and the sensor coordinates. Then, the homography matrix is ​​decomposed by the singular value decomposition algorithm to obtain the matrix decomposition result, and the positional relationship of the matrix decomposition result is logically screened, and then the external parameters of the binocular vision sensor are obtained according to the logical screening result.

[0124] Step S509: determining the cost function of the external parameters through the internal parameter matrix, the radial distortion factor, the sensor distribution parameter of the binocular vision sensor and the image coordinates of each feature point matching pair;

[0125] Step S510: adjusting the external parameters according to the cost function to obtain optimized external parameters, and using the optimized external parameters as the calibration result of the binocular vision sensor.

[0126] After obtaining the external parameters of the binocular vision sensor, the external parameters are nonlinearly optimized to improve the accuracy of the binocular vision sensor calibration result. The cost function can be determined according to the cumulative projection error of the feature point matching pairs corresponding to each cross laser line image group.

[0127] It should be understood that although Figure 2-3 The steps in the flowchart of , 5 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-3 At least part of the steps in 5 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0128] In one embodiment, Figure 6 As shown, a binocular vision sensor calibration device is provided, including: an image group acquisition module 601, an intersection determination module 603, a matching pair determination module 605, a mapping matrix determination module 607 and a calibration result acquisition module 609, wherein:

[0129] An image group acquisition module 601 is used to acquire each cross laser line image group captured by the binocular vision sensor to be calibrated under different shooting conditions, wherein the cross laser line image group includes a left cross laser line image and a right cross laser line image;

[0130] The intersection point determination module 603 is used to determine the intersection point of the left image of the cross laser line in the left cross laser line image, and determine the intersection point of the right image of the cross laser line in the right cross laser line image;

[0131] A matching pair determination module 605 is used to obtain a matching pair of feature points corresponding to the cross laser line image group according to the intersection points of the left image and the intersection points of the right image;

[0132] A mapping matrix determination module 607 is used to determine the position mapping matrix between the left cross laser line image and the right cross laser line image according to each feature point matching pair and the internal parameters of the binocular vision sensor;

[0133] The calibration result acquisition module 609 is used to decompose the position mapping matrix to obtain the external parameters of the binocular vision sensor. The calibration result of the binocular vision sensor includes the external parameters.

[0134] In one embodiment, the image group acquisition module 601 includes an image group acquisition unit, which is used to adjust the position of a preset cross laser when the binocular vision sensor to be calibrated is in different sensor distribution parameters, and obtain each cross laser line image group respectively captured by the binocular vision sensor.

[0135] In one embodiment, the intersection determination module 603 includes a filtering and denoising unit and an intersection determination unit; wherein the filtering and denoising unit is used to filter the left cross laser line image and the right cross laser line image respectively to obtain the left denoised image and the right denoised image; the intersection determination unit is used to perform laser straight line fitting on the cross laser lines in the left denoised image and the right denoised image, and determine the intersection points of the left image and the right image respectively according to the straight line fitting results.

[0136] In one embodiment, the intrinsic parameters of the binocular vision sensor include an internal parameter matrix; the position mapping matrix includes a homography matrix; the mapping matrix determination module 607 includes a mapping function unit, a sensor coordinate unit and a matrix determination unit; wherein the mapping function unit is used to determine the sensor coordinate mapping function between the feature points in each feature point matching pair; the sensor coordinate unit is used to determine the sensor coordinates corresponding to the feature point according to the image coordinates of the feature point and the internal parameter matrix, and the sensor coordinates are the coordinates of the feature point in the corresponding sensor coordinate system; the matrix determination unit is used to obtain the homography matrix between the left cross laser line image and the right cross laser line image according to the sensor coordinate mapping function and the sensor coordinates.

[0137] In one of the embodiments, it also includes an internal parameter determination module for performing internal parameter calibration processing on the binocular vision sensor to obtain the internal parameters of the binocular vision sensor.

[0138] In one embodiment, the calibration result acquisition module 609 includes a matrix decomposition unit and a result screening unit; wherein the matrix decomposition unit is used to decompose the homography matrix through a singular value decomposition algorithm to obtain a matrix decomposition result; the result screening unit is used to perform logical screening on the positional relationship of the matrix decomposition result, and obtain the external parameters of the binocular vision sensor based on the logical screening result.

[0139] In one of the embodiments, the internal parameters of the binocular vision sensor also include a radial distortion factor; it also includes a cost function module and a result optimization module; wherein the cost function module is used to determine the cost function of the external parameters through the internal parameter matrix, the radial distortion factor, the sensor distribution parameters of the binocular vision sensor and the image coordinates of each feature point matching pair; the result optimization module is used to adjust the external parameters according to the cost function to obtain the optimized external parameters, and use the optimized external parameters as the calibration results of the binocular vision sensor.

[0140] For the specific definition of the binocular vision sensor calibration device, please refer to the definition of the binocular vision sensor calibration method above, which will not be repeated here. Each module in the above binocular vision sensor calibration device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0141] In one embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a binocular vision sensor calibration method is implemented.

[0142] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0144] Acquire each cross laser line image group respectively captured by the binocular vision sensor to be calibrated under different shooting conditions, the cross laser line image group including a left cross laser line image and a right cross laser line image;

[0145] Determine a left image intersection point of the cross laser line in the left cross laser line image, and determine a right image intersection point of the cross laser line in the right cross laser line image;

[0146] According to the intersection points of the left image and the right image, a matching pair of feature points corresponding to the cross laser line image group is obtained;

[0147] Determine the position mapping matrix between the left cross laser line image and the right cross laser line image according to the matching pairs of feature points and the internal parameters of the binocular vision sensor;

[0148] The position mapping matrix is ​​decomposed to obtain the external parameters of the binocular vision sensor. The calibration result of the binocular vision sensor includes the external parameters.

[0149] In one embodiment, when the processor executes the computer program, the following steps are also implemented: when the binocular vision sensor to be calibrated is in different sensor distribution parameters, the position of the preset cross laser is adjusted to obtain each cross laser line image group respectively captured by the binocular vision sensor.

[0150] In one embodiment, when the processor executes the computer program, the following steps are also implemented: filtering the left cross laser line image and the right cross laser line image respectively to obtain the left denoised image and the right denoised image; performing laser straight line fitting on the cross laser lines in the left denoised image and the right denoised image, and determining the intersection points of the left image and the right image respectively according to the straight line fitting results.

[0151] In one embodiment, the internal parameters of the binocular vision sensor include an internal parameter matrix; the position mapping matrix includes a homography matrix; when the processor executes the computer program, the following steps are also implemented: determining the sensor coordinate mapping function between the feature points in each feature point matching pair; determining the sensor coordinates corresponding to the feature points according to the image coordinates of the feature points and the internal parameter matrix, the sensor coordinates are the coordinates of the feature points in the corresponding sensor coordinate system; according to the sensor coordinate mapping function and the sensor coordinates, obtaining the homography matrix between the left cross laser line image and the right cross laser line image.

[0152] In one embodiment, when the processor executes the computer program, the following steps are also implemented: performing internal parameter calibration processing on the binocular vision sensor to obtain internal parameters of the binocular vision sensor.

[0153] In one embodiment, when the processor executes the computer program, the following steps are also implemented: decomposing the homography matrix through a singular value decomposition algorithm to obtain a matrix decomposition result; performing logical screening on the positional relationship of the matrix decomposition result, and obtaining the external parameters of the binocular vision sensor based on the logical screening result.

[0154] In one embodiment, the intrinsic parameters of the binocular vision sensor also include a radial distortion factor; when the processor executes the computer program, the following steps are also implemented: the cost function of the extrinsic parameters is determined through the internal parameter matrix, the radial distortion factor, the sensor distribution parameters of the binocular vision sensor and the image coordinates of each feature point matching pair; the external parameters are adjusted according to the cost function to obtain optimized external parameters, and the optimized external parameters are used as the calibration results of the binocular vision sensor.

[0155] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0156] Acquire each cross laser line image group respectively captured by the binocular vision sensor to be calibrated under different shooting conditions, the cross laser line image group including a left cross laser line image and a right cross laser line image;

[0157] Determine a left image intersection point of the cross laser line in the left cross laser line image, and determine a right image intersection point of the cross laser line in the right cross laser line image;

[0158] According to the intersection points of the left image and the right image, a matching pair of feature points corresponding to the cross laser line image group is obtained;

[0159] Determine the position mapping matrix between the left cross laser line image and the right cross laser line image according to the matching pairs of feature points and the internal parameters of the binocular vision sensor;

[0160] The position mapping matrix is ​​decomposed to obtain the external parameters of the binocular vision sensor. The calibration result of the binocular vision sensor includes the external parameters.

[0161] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the binocular vision sensor to be calibrated is in different sensor distribution parameters, the position of the preset cross laser is adjusted to obtain each cross laser line image group respectively captured by the binocular vision sensor.

[0162] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: filtering the left cross laser line image and the right cross laser line image respectively to obtain the left denoised image and the right denoised image; performing laser straight line fitting on the cross laser lines in the left denoised image and the right denoised image, and determining the intersection points of the left image and the right image respectively according to the straight line fitting results.

[0163] In one embodiment, the intrinsic parameters of the binocular vision sensor include an internal parameter matrix; the position mapping matrix includes a homography matrix; when the computer program is executed by the processor, the following steps are also implemented: determining the sensor coordinate mapping function between the feature points in each feature point matching pair; determining the sensor coordinates corresponding to the feature points according to the image coordinates of the feature points and the internal parameter matrix, the sensor coordinates are the coordinates of the feature points in the corresponding sensor coordinate system; according to the sensor coordinate mapping function and the sensor coordinates, obtaining the homography matrix between the left cross laser line image and the right cross laser line image.

[0164] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing internal parameter calibration processing on the binocular vision sensor to obtain the internal parameters of the binocular vision sensor.

[0165] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: decomposing the homography matrix through a singular value decomposition algorithm to obtain a matrix decomposition result; performing logical screening on the positional relationship of the matrix decomposition result, and obtaining the external parameters of the binocular vision sensor based on the logical screening result.

[0166] In one embodiment, the intrinsic parameters of the binocular vision sensor also include a radial distortion factor; when the computer program is executed by the processor, the following steps are also implemented: the cost function of the extrinsic parameters is determined through the internal parameter matrix, the radial distortion factor, the sensor distribution parameters of the binocular vision sensor and the image coordinates of each feature point matching pair; the external parameters are adjusted according to the cost function to obtain optimized external parameters, and the optimized external parameters are used as the calibration results of the binocular vision sensor.

[0167] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0168] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A binocular vision sensor calibration method, the method comprising: Acquire each cross laser line image group respectively captured by the binocular vision sensor to be calibrated under different shooting conditions, wherein the cross laser line image group includes a left cross laser line image and a right cross laser line image; Determine a left image intersection point of the cross laser line in the left cross laser line image, and determine a right image intersection point of the cross laser line in the right cross laser line image; Obtaining a feature point matching pair corresponding to the cross laser line image group according to the intersection points of the left image and the intersection points of the right image; Determining a sensor coordinate mapping function between feature points in each of the feature point matching pairs; Determine the sensor coordinates corresponding to the feature point according to the image coordinates of the feature point and the internal parameter matrix included in the internal parameters of the binocular vision sensor, wherein the sensor coordinates are the coordinates of the feature point in the corresponding sensor coordinate system; Obtaining a homography matrix between the left cross laser line image and the right cross laser line image according to the sensor coordinate mapping function and the sensor coordinates; The homography matrix is ​​decomposed to obtain the extrinsic parameters of the binocular vision sensor, and the calibration result of the binocular vision sensor includes the extrinsic parameters.

2. The method according to claim 1, characterized in that The acquisition of each cross laser line image group obtained by taking pictures respectively by the binocular vision sensor to be calibrated under different shooting conditions comprises: When the binocular vision sensor to be calibrated is in different sensor distribution parameters, the position of the preset cross laser is adjusted to obtain each cross laser line image group respectively photographed by the binocular vision sensor.

3. The method according to claim 1, characterized in that The determining of the left image intersection point of the cross laser line in the left cross laser line image and the determining of the right image intersection point of the cross laser line in the right cross laser line image comprises: Performing filtering processing on the left cross laser line image and the right cross laser line image respectively to obtain a left denoised image and a right denoised image; Laser straight line fitting is performed on the cross laser lines in the left denoised image and the right denoised image, and the intersection point of the left image and the intersection point of the right image are determined respectively according to the straight line fitting results.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: An internal parameter calibration process is performed on the binocular vision sensor to obtain the internal parameters of the binocular vision sensor.

5. The method according to claim 1, characterized in that Decomposing the homography matrix to obtain the external parameters of the binocular vision sensor includes: Decomposing the homography matrix by a singular value decomposition algorithm to obtain a matrix decomposition result; The positional relationship of the matrix decomposition result is logically screened, and the external parameters of the binocular vision sensor are obtained according to the logical screening result.

6. The method according to claim 1, characterized in that The internal parameters of the binocular vision sensor also include radial distortion factors; After decomposing the homography matrix to obtain the external parameters of the binocular vision sensor, the method further includes: Determining the cost function of the external parameters through the internal parameter matrix, the radial distortion factor, the sensor distribution parameter of the binocular vision sensor and the image coordinates of each of the feature point matching pairs; The extrinsic parameters are adjusted according to the cost function to obtain optimized extrinsic parameters, and the optimized extrinsic parameters are used as calibration results of the binocular vision sensor.

7. A binocular vision sensor calibration device, characterized in that: The device comprises: An image group acquisition module is used to acquire each cross laser line image group respectively captured by the binocular vision sensor to be calibrated under different shooting conditions, wherein the cross laser line image group includes a left cross laser line image and a right cross laser line image; An intersection point determination module, used to determine the intersection point of the left image of the cross laser line in the left cross laser line image, and determine the intersection point of the right image of the cross laser line in the right cross laser line image; A matching pair determination module, used for obtaining a matching pair of feature points corresponding to the cross laser line image group according to the intersection points of the left image and the intersection points of the right image; A mapping matrix determination module is used to determine a sensor coordinate mapping function between feature points in each of the feature point matching pairs; determine the sensor coordinates corresponding to the feature points according to the image coordinates of the feature points and the internal parameter matrix included in the internal parameters of the binocular vision sensor, wherein the sensor coordinates are the coordinates of the feature points in the corresponding sensor coordinate system; obtain a homography matrix between the left cross laser line image and the right cross laser line image according to the sensor coordinate mapping function and the sensor coordinates; The calibration result acquisition module is used to decompose the homography matrix to obtain the external parameters of the binocular vision sensor. The calibration result of the binocular vision sensor includes the external parameters.

8. The device according to claim 7, characterized in that The image group acquisition module is also used to adjust the position of the preset cross laser when the binocular vision sensor to be calibrated is in different sensor distribution parameters, and obtain each cross laser line image group respectively photographed by the binocular vision sensor.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images