A vision calibration method for a weeding mechanism based on a laser range finder

Abstract

The present application relates to the field of visual calibration of weeding mechanism, and particularly relates to a visual calibration method of weeding mechanism based on laser range finder, which comprises the following steps: 1, establishing a visual calibration system; 2, calculating the pose of the planar coordinate system to the camera coordinate system; 3, calculating the planar parameter of the calibration plane in the camera coordinate system; 4, calculating the pose of the laser coordinate system to the camera coordinate system; 5, calculating the two-dimensional pixel coordinates of the weeding mechanism in the pixel coordinate system. The visual calibration method of weeding mechanism based on laser range finder determines the two-dimensional pixel coordinates of the weeding mechanism, and solves the determination of the two-dimensional coordinate position of the weeding mechanism when the weeding mechanism is not in the camera field of view or the coordinates of the weeding mechanism relative to the vehicle body are unknown.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual calibration of weeding mechanism, and particularly relates to a visual calibration method of weeding mechanism based on laser range finder. BACKGROUND

[0002] The problem of seedling damage caused by crop row bending seriously restricts the development of mechanical weeding technology. Therefore, according to the information of crop row, it is an urgent problem to be solved for mechanical weeding to realize row seedling avoidance.

[0003] At present, the position information of the weeding mechanism is obtained by using RTK GPS to obtain the absolute position of the weeding device, or by converting the image coordinate system to the ground coordinate system in real time. This method needs to know the coordinates of the weeding mechanism relative to the vehicle body in advance.

[0004] The present application provides a visual calibration method of weeding mechanism based on laser range finder to realize real-time positioning of the two-dimensional coordinates of the weeding mechanism in the pixel coordinate system, so as to calculate the lateral pixel deviation of the weeding mechanism and the crop row guide line, and compensate for the deviation. SUMMARY

[0005] In order to solve the problems mentioned in the background, the present application aims to provide a visual calibration method of weeding mechanism based on laser range finder.

[0006] The purpose of the present application can be realized by the following technical solutions:

[0007] A visual calibration method of weeding mechanism based on laser range finder, the recognition method comprises the following steps:

[0008] I. Establish a visual calibration system;

[0009] II. Calculate the pose of the plane coordinate system to the camera coordinate system;

[0010] III. Calculate the plane parameters of the calibration plane in the camera coordinate system;

[0011] IV. Calculate the pose of the laser coordinate system to the camera coordinate system;

[0012] V. Calculate the two-dimensional pixel coordinates of the weeding mechanism in the pixel coordinate system.

[0013] Further, the specific steps of step I are as follows:

[0014] A1. Establish a visual calibration system including a calibration plane, a laser range finder, a camera and a weeding mechanism, wherein the laser range finder and the weeding mechanism are fixed together, and the camera is fixed;

[0015] A2. The calibration plane distance laser rangefinder moves along the z-axis within a range of 2m. At each position on the z-axis, the calibration plane is rotated around the x-axis and y-axis to ensure that the laser point falls within the calibration plane.

[0016] A3. Record the distance from the calibration plane to the laser rangefinder and acquire an image of the calibration plane.

[0017] Furthermore, the specific steps of step two are as follows:

[0018] B1. First, calibrate the camera to obtain its intrinsic parameter matrix.

[0019] B2. The homography matrix H = K[RT] between the calibration plane and the image plane is calculated using the DLT (Direct Linear Transform) algorithm, where R and T are the rotation and translation matrices from the calibration plane coordinate system to the camera coordinate system, respectively.

[0020] B3. The pose from the calibration plane coordinate system to the camera coordinate system is calculated based on the homography matrix and the camera's intrinsic parameter matrix as follows:

[0021] [RT] = K -1 H (7)

[0022] Furthermore, the specific steps of step three are as follows:

[0023] C1. Assume the calibration plane is located in the camera coordinate system using (n) C d C )express;

[0024] C2, n C To determine the normal vector of the calibration plane in the camera coordinate system, based on n W = [0, 0, 1] T Given the rotation matrix R of the calibration plane coordinate system and the camera coordinate system, the normal vector n of the calibration plane can be calculated. C as follows:

[0025] n C =R*n W (8)

[0026] C3, P C To determine the coordinates of the origin of the calibration plane in the camera coordinate system, P C =T, according to The parameter d is calculated. C as follows:

[0027] d C =-R*n W *T (9)

[0028] Further, the specific steps of the step four are as follows:

[0029] D1, according to the laser point on the calibration plane, it is known that Put n C = [n 11 , n 21 , n 31 ] T , Substitute, can get

[0030]

[0031] Convert formula (10) to a multivariate linear equation AX=b, where A=[n 11 l, n 21 l, n 31 l, n 11 , n 21 , n 31 ], X=[R 13 , R 23 , R 33 , t 11 , t 21 , t 31 ] T , b=[-d C ], since is an orthogonal matrix, it satisfies the constraint condition that the modulus of the vector is 1, so X satisfies the constraint condition of formula (11):

[0032] R 13 2 +R 23 2 +R 33 2 =1 (11)

[0033] D2, since the analytical solution of the multivariate nonlinear equation cannot be solved, formula (10) is converted into a nonlinear optimization problem of multivariate variables, and the optimization function is established as follows:

[0034]

[0035] D3, first, solve the optimization function without nonlinear constraints, and the solved X is used as the initial value of the nonlinear optimization problem, and then the optimization problem is iteratively solved based on the initial value.

[0036] Further, the specific steps of the step five are as follows:

[0037] E1, the weeding mechanism and the laser range finder are fixed together, the coordinates of the weeding mechanism in the laser coordinate system are defined as the origin (0, 0, 0) of the laser coordinate system, according to the coordinate transformation relationship between the laser coordinate system and the camera coordinate system, is the three-dimensional coordinate of the weeding mechanism in the camera coordinate system.

[0038] E2, the three-dimensional coordinate P(X, Y, Z) in the camera coordinate system is transformed into the pixel coordinate system based on the camera intrinsic matrix to obtain the coordinate P(u, v), as shown in the following formula:

[0039]

[0040] The beneficial effects of the present application are:

[0041] The present application determines the two-dimensional pixel coordinates of the weeding mechanism based on the visual calibration method of the laser range finder weeding mechanism, and solves the determination of the two-dimensional coordinate position of the weeding mechanism when the weeding mechanism is not in the camera field of view or the coordinate of the weeding mechanism relative to the vehicle body is unknown.

[0042] By determining the two-dimensional coordinate information of the weeding mechanism, the present application can compensate for the lateral position deviation of the crop row and the weeding mechanism based on the crop row position information, and solve the row-avoiding-sprout problem in mechanical weeding. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description, and obviously, other drawings can also be obtained by those skilled in the art without creative labor on the premise of these drawings.

[0044] Figure 1 is the flow chart of the visual calibration method of the weeding mechanism based on the laser range finder of the present application;

[0045] Figure 2 is the two-dimensional coordinate of the weeding mechanism in the pixel coordinate system determined by the visual calibration method of the weeding mechanism based on the laser range finder of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0047] A visual calibration method of a weeding mechanism based on a laser range finder, as shown in Figure 1 , includes the following steps:

[0048] I. Establish a visual calibration system;

[0049] II. Pose calculation of the plane coordinate system to the camera coordinate system

[0050] III. Plane parameter calculation of the calibration plane in the camera coordinate system

[0051] IV. Pose calculation of the laser coordinate system to the camera coordinate system

[0052] V. Two-dimensional pixel coordinate calculation of the weeding mechanism in the pixel coordinate system

[0053] Wherein, the establishment of the vision calibration system comprises the following steps:

[0054] A1, establishing a vision calibration system comprising a calibration plane, a laser range finder, a camera and a weeding mechanism, wherein the laser range finder and the weeding mechanism are fixed together, and the camera is fixed;

[0055] A2, moving the calibration plane along the z-axis within a range of 2m from the laser range finder, and at each position of the z-axis, rotating the calibration plane around the x-axis and the y-axis to ensure that the laser point falls within the calibration plane;

[0056] A3, recording the distance from the calibration plane to the laser range finder and collecting the calibration plane image.

[0057] Wherein, the pose calculation of the plane coordinate system to the camera coordinate system comprises the following steps:

[0058] B1, first calibrate the camera to obtain the intrinsic matrix of the camera

[0059] B2, using the DLT (Direct Linear Transform) algorithm to calculate the homography matrix H = K [RT] of the calibration plane and the image plane, wherein R and T are the rotation matrix and the translation matrix of the calibration plane coordinate system to the camera coordinate system, respectively;

[0060] B3, calculating the pose of the calibration plane coordinate system to the camera coordinate system according to the homography matrix and the intrinsic matrix of the camera as follows:

[0061] [R T] = K -1 H (14)

[0062] Wherein, the plane parameter calculation of the calibration plane in the camera coordinate system comprises the following steps:

[0063] C1, assuming that the calibration plane in the camera coordinate system is represented by (n C , d C );

[0064] C2, n C is the normal vector of the calibration plane in the camera coordinate system, and according to n W= [0, 0, 1] T and the rotation matrix R of the calibration plane coordinate system and the camera coordinate system, the normal vector n of the calibration plane can be calculated C As follows:

[0065] n C = R * n W (15)

[0066] C3, P C is the coordinate of the origin of the calibration plane in the camera coordinate system, P C = T, according to The parameter d is calculated C As follows:

[0067] d C = -R * n W *T (16)

[0068] Wherein, the pose calculation of the laser coordinate system to the camera coordinate system includes the following steps:

[0069] D1, according to the laser point on the calibration plane, it is known that Put n C = [n 11 , n 21 , n 31 ] T , Substitute, can get

[0070]

[0071] Transform equation (17) into a multivariate linear equation AX = b, where A = [n 11 l, n 21 l, n 31 l, n 11 , n 21 , n 31 ], X = [R 13 , R 23 , R 33 , t 11 , t 21 , t 31 ] T , b = [-d C ]; Since is an orthogonal matrix, it satisfies the constraint condition that the modulus of the vector is 1, so X satisfies the constraint condition of equation (18):

[0072] R 13 2 + R 23 2 + R 33 2= 1 (18)

[0073] D2, formula (17) is a multivariate nonlinear equation, because the analytical solution of the multivariate nonlinear equation cannot be solved, formula (17) is converted into a nonlinear optimization problem of multivariate, and the optimization function is established as follows:

[0074]

[0075] D3, the optimization function without nonlinear constraint is solved, and the solved X is used as the initial value of the nonlinear optimization problem, and then the optimization problem is iteratively solved based on the initial value. The initial value of the nonlinear optimization function is calculated as follows:

[0076] X0= (A T A) -1 A T b (20)

[0077] D4, the nonlinear optimization problem with constraint is converted into a nonlinear optimization problem without constraint, and the specific modeling is as follows:

[0078]

[0079] D5, formula (21) is solved by iterative optimization based on the initial point X0.

[0080] Wherein, the specific steps of calculating the two-dimensional pixel coordinates of the weeding mechanism in the pixel coordinate system are as follows:

[0081] E1, the weeding mechanism and the laser range finder are fixed together, the coordinates of the weeding mechanism in the laser coordinate system are defined as the origin (0, 0, 0) of the laser coordinate system, according to the coordinate transformation relationship between the laser coordinate system and the camera coordinate system, That is, the three-dimensional coordinates of the weeding mechanism in the camera coordinate system.

[0082] E2, the three-dimensional coordinates P (X, Y, Z) in the camera coordinate system are transformed into the pixel coordinate system based on the camera intrinsic matrix, and the coordinates P (u, v) can be obtained, as shown in the following formula:

[0083]

[0084] Embodiment:

[0085] The establishment of visual calibration hardware system includes the transmission belt simulation weed removal mechanism running, the middle part of the transmission belt is provided with a rack, the side of the transmission belt is provided with a computer, the rack is provided with an Intel Realsense D435 camera and a laser range finder fixedly connected with the weed removal mechanism above, and the weed removal mechanism can be controlled to move through an electric push rod. In order to pursue high precision, 25 two-dimensional code calibration plane poses are captured at random positions, and the projection point of the laser range finder is ensured to fall on the calibration plane.

[0086] We use two methods to evaluate the results of calibration. Method 1 is to compare the 2-norm of the deviation between the actual position of the laser range finder in the camera coordinate system and the calibrated position, and the results are shown in Table 1:

[0087] Table 1 results of calibration position deviation of method 1

[0088]

[0089] As can be seen from Table 1, when the distance deviation between the actual distance and the calibrated distance of the laser range finder and the camera is calculated using 10, 12, 14, 16, 18, 20, 22, 24 and 25 laser projection point images, the distance deviation results of the first group of data are 1.845 cm, 1.499 cm, 1.473 cm, 0.864 cm, 0.622 cm, 0.689 cm, 0.730 cm and 0.100 cm, and the distance deviation of the second group of data is 2.843 cm, 3.116 cm, 1.485 cm, 1.376 cm, 0.246 cm, 0.0642 cm, 0.222 cm and 0.100 cm. With the increase of the number of projection point images, the distance deviation of the first group and the second group of data shows a downward trend as a whole, and the distance deviation of the third group of data fluctuates greatly. From the results of the average distance deviation, the average distance deviation of the three groups of randomly arranged data shows a downward trend as a whole. When 25 images are used for calibration, the average distance deviation reaches the minimum of 0.100 cm.

[0090] Method 2 is to compare the average value of the deviation between the theoretically calculated projection point of the laser range finder on the calibration plate and the actual projection point, and the results are shown in Table 2:

[0091] Table 2 results of projection point pixel deviation of method 2

[0092]

[0093] When the coordinate transformation parameters from the laser coordinate system to the camera coordinate system are calculated using 10, 12, 14, 16, 18, 20, 22, 24 and 25 laser projection point images, the pixel deviation results of the three groups of data all show a clear downward trend. According to Table 2, the average pixel deviation results of the three groups of data are 270.999, 237.107, 200.842, 167.027, 133.662, 99.642, 66.181, 27.923 and 9.412 pixels, respectively. Therefore, with the increase of the number of projection point images, the pixel deviation of the projection point rapidly decreases, indicating that the accuracy of the coordinate transformation parameters is improved. When 25 images are used, the minimum pixel deviation of the projection point is only 9.412 pixels.

[0094] The application determines the two-dimensional pixel coordinates of the weeding mechanism based on the vision calibration method of the laser range finder, and solves the determination of the two-dimensional coordinate position of the weeding mechanism when the weeding mechanism is not in the field of view of the camera or the coordinates of the weeding mechanism relative to the vehicle body are unknown.

[0095] By determining the two-dimensional coordinate information of the weeding mechanism, the transverse position deviation of the crop row and the weeding mechanism can be compensated based on the crop row position information, and the row-avoiding-sprout problem in mechanical weeding is solved. The above shows and describes the basic principles, main features and advantages of the application. It should be understood by those skilled in the art that the application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the application. Without departing from the spirit and scope of the application, various changes and improvements can be made to the application, and these changes and improvements all fall within the scope of the claimed application.

Claims

1.A method for vision calibration of a weeding mechanism based on a laser range finder, characterized in that, The calibration method comprises the following steps: I. Establishing a visual calibration system; II. Calculating the pose of the planar coordinate system to the camera coordinate system; III. Calculating the planar parameters of the calibration plane in the camera coordinate system; IV. Calculating the pose of the laser coordinate system to the camera coordinate system based on the planar parameters; V. Obtaining the two-dimensional pixel coordinates of the weeding mechanism in the pixel coordinate system based on the pose of the laser coordinate system to the camera coordinate system; The specific steps of step I are as follows: A1. Establishing a visual calibration system comprising a calibration plane, a laser range finder, a camera and a weeding mechanism, wherein the laser range finder and the weeding mechanism are fixed together, and the camera is fixed; A2. Moving the calibration plane along the z-axis within a range of 2m from the laser range finder, rotating the calibration plane around the x-axis and the y-axis at each position of the z-axis, and ensuring that the laser point falls within the calibration plane; A3. Recording the distance from the calibration plane to the laser range finder and collecting the image of the calibration plane; The specific steps of step II are as follows: B1. First, calibrate the camera to obtain the intrinsic matrix K of the camera ; B2, a DLT (Direct Linear Transform) algorithm is used to calculate the homography matrix between the calibration plane and the image plane wherein R, T are the rotation matrix and translation matrix from the calibration plane coordinate system to the camera coordinate system, respectively B3. Calculating the pose of the planar coordinate system to the camera coordinate system according to the homography matrix and the intrinsic matrix of the camera as follows: = （1）； The specific steps of step V are as follows: E1, the weeding mechanism and the laser range finder are fixed together, the coordinates of the weeding mechanism in the laser coordinate system are defined as the origin (0, 0, 0) of the laser coordinate system, and the three-dimensional coordinates of the weeding mechanism in the camera coordinate system are obtained according to the coordinate transformation relationship between the laser coordinate system and the camera coordinate system ; E2. Transform the three-dimensional coordinates P in the camera coordinate system into the pixel coordinate system based on the camera intrinsic matrix As shown in the following formula:​ (6）。 2. The vision calibration method for a weeding mechanism based on a laser range finder according to claim 1, wherein, The specific steps of step III are as follows: C1, assuming that the calibration plane is represented in the camera coordinate system by C1= (0, 0, 1)T. C2、 To calibrate the normal vector of the plane in the camera coordinate system, according to and the rotation matrix R of the calibration plane coordinate system and the camera coordinate system, the normal vector of the calibration plane is calculated as follows: （2） C3、 the coordinates of the origin of the calibration plane in the camera coordinate system, , according to the calculated parameters （3）。

Images