Pixel coordinate and galvanometer scanner offset mapping method based on laser point detection and application

Abstract

The invention belongs to a computer vision technology, mainly relates to the field of laser weeding, and particularly relates to a pixel coordinate and galvanometer scanner offset mapping method based on laser point detection and application. The method is mainly used in the field of laser weeding, and is used for calculating galvanometer offset through weed pixel position information acquired by a camera and controlling laser to accurately reach a weed point. The invention provides a method for directly mapping a target from pixel coordinates to galvanometer offset. According to tests, the average precision error of the method is within 2 mm, the maximum error does not exceed 5 mm, the error does not exceed 3 mm under the 88% condition, and the requirement for precise weeding can be met.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a pixel coordinate and galvanometer scanner offset mapping method based on laser point detection and its application. Background Art

[0002] In the agricultural field, good weed control has always been one of the important factors affecting crop growth. However, traditional manual weeding is not only costly, but also has uncontrollable effects.

[0003] In recent years, with the maturity of deep learning technology in the field of vision, vision-based automatic weeding technology has received more and more attention.

[0004] Automatic weeding technology is usually divided into three categories: mechanical, chemical, and laser. The mechanical hoe for mechanical weeding is complex in design and easily causes damage to crops; chemical weeding is easy to cause environmental pollution and bring unpredictable future costs; laser weeding hoes are simpler than mechanical hoes and have almost no pollution to the environment. Compared with the other two methods, its disadvantage is that it requires higher weed positioning accuracy.

[0005] At present, there are relevant patents and technologies for laser-based weeding methods and equipment, but these patents and technologies often lack detailed mapping calculation instructions. The few that have mapping instructions also control the rotation angle of the galvanometer, but this limits the scope of use of the method because many galvanometers only provide control interfaces for x and y axis offsets and cannot be operated by angle. Summary of the invention

[0006] The invention discloses a method for mapping pixel coordinates and galvanometer scanner (hereinafter referred to as galvanometer) offset based on laser point detection. The invention is mainly used in the field of laser weeding, and is used to calculate the galvanometer offset through the weed pixel position information obtained by the camera, and control the laser to accurately reach the weed point.

[0007] In order to solve the above technical problems, we propose a pixel coordinate and galvanometer scanner offset mapping method and application based on laser point detection.

[0008] The present invention provides the following technical solutions:

[0009] A pixel coordinate and galvanometer scanner offset mapping method based on laser point detection, by performing a series of coordinate system transformations on p(x,y), gm_dist(x,y)' is obtained.

[0010] Among them, p(x,y) is the pixel coordinate system, gm_dist(x,y)' is the distortion offset calculated by the formula, and the specific steps are:

[0011] S100, using the depth camera parameters, convert the pixel coordinate system p(x,y) to the camera coordinate system c(x,y,z):

[0012] c(x,y,z)=F p→c (p(x,y)),

[0013] Among them, F p→c is the mapping function from pixel coordinates to camera coordinate system, z represents the depth distance of the target from the camera,

[0014] S200, convert the target from the camera coordinate system c(x,y,z) to the galvanometer scanner coordinate system g(x,y,z):

[0015]

[0016] Among them, X, Y, and Z are the deviations between the midpoint of the camera coordinate system and the midpoint of the galvanometer coordinate system in the physical coordinate system.

[0017] S300, convert the target from the galvanometer coordinate system g(x,y,z) into the galvanometer offset gm(x,y):

[0018]

[0019] Among them, f x , f y is the scale factor, G x , G y is the deviation between the midpoint of the galvanometer offset obtained by the pinhole camera model and the accurate midpoint of the galvanometer offset.

[0020] S400, applying distortion to the galvanometer offset gm(x,y) to obtain the final distortion offset gm_dist(x,y)':

[0021]

[0022] rr=(gm x -G x ) 2 +(gm y -G y ) 2

[0023]

[0024] Among them, X i : Distortion polynomial coefficient of the x-axis, Y 1 : y-axis radial distortion parameter,

[0025] S500, construct the loss function between gm_dist(x,y) and gm_dist(x,y)', and obtain all parameters in the formula according to the least square method, where gm_dist(x,y) is the randomly sampled galvanometer offset, which is the accurate actual distortion offset, and gm_dist(x,y)' is the distortion offset calculated by the formula, and the goal is to be as close as possible to gm_dist(x,y);

[0026] S600, set the parameters X, Y, Z, f x 、f y , G x , G y , X 0 ~X n and Y 1 Substitute into the above S100~S400 formulas and obtain gm_dist(x,y)' according to p(x,y).

[0027] Preferably, in the above S400, the distortion polynomial coefficients of the x-axis are fitted by a third-order polynomial.

[0028] Preferably, the acquisition of the p(x, y) and gm_dist(x, y) sample data uses a laser lamp as a light source and combines it with a template matching method to obtain N groups (p(x, y), gm_dist(x, y)).

[0029] An application of a method for mapping pixel coordinates and galvanometer scanner offsets based on laser point detection, running the aforementioned method for mapping pixel coordinates and galvanometer scanner offsets based on laser point detection, is used for any scenario where the laser is controlled to operate through camera data.

[0030] A laser weeding robot, comprising:

[0031] Memory for storing computer programs;

[0032] A processor is used to implement the aforementioned pixel coordinate and galvanometer scanner offset mapping method based on laser point detection when executing the computer program.

[0033] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the pixel coordinate and galvanometer scanner offset mapping method based on laser point detection as described above is implemented.

[0034] Compared with the prior art, the invention has the following beneficial effects: an automatic high-precision mapping method of target pixels and galvanometer offset is proposed. The method can be used not only for weeding, but also for any scenario where the laser is controlled by camera data.

[0035] The present invention proposes a method for mapping pixel coordinates and galvanometer offset based on laser point detection. The method cleverly uses laser light as a light source for data acquisition; constructs a coordinate system mapping formula through a series of analytical modeling; and uses the least squares method for parameter estimation. Thus, the pixel position of the target can be accurately converted into a galvanometer offset, thereby controlling the laser for precise weeding.

[0036] The method proposed by the present invention can be fully automatically operated, and can accurately obtain such mapping relationships without additional human processing, and is easy to update. In experimental statistics, the average absolute error is <2mm, the maximum is <5mm, and in 88% of cases it is <3mm; in actual tests, the same accuracy as in the experiment is also observed, indicating that the present invention can meet the requirements of precision weed control. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0038] Figure 1 It is a schematic diagram of the process of the present invention;

[0039] Figure 2 It is a map of the relative positions of the depth camera, galvanometer scanner, and laser;

[0040] Figure 3 It is a laser light imaging diagram;

[0041] Figure 4 is the galvanometer offset gm x and gm y Sampling point distribution map;

[0042] Figure 5 is the galvanometer coordinate system and Sampling point distribution map;

[0043] Figure 6 This is an illustration from the paper "Analysis and Correction of Distortion Mechanism of MEMS Galvanometer Scanning Confocal Images"

[0044] Figure 7 The x-axis is modeled by a cubic polynomial function. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] 1 Overall idea

[0047] In the present invention, a depth camera capable of acquiring depth information of an object is used, and the depth camera and the galvanometer are fixed together, such as Figure 2 As shown:

[0048] The pixel coordinates p(x,y) of the target midpoint are obtained through the color image of the depth camera and then processed by a mapping function F p→gm_dist , complete the conversion to the galvanometer offset gm_dist(x,y)', the overall formula is:

[0049] gm_dist(x,y)'=F p→gm_dist (p(x,y)) (1)

[0050] In the subsequent content of the present invention, the acquisition of this mapping relationship will be described in detail.

[0051] 2 Data collection

[0052] The purpose of data acquisition is to obtain multiple sets of target coordinates p(x, y) and galvanometer offset gm_dist(x, y) data in the camera pixel coordinate system as the data source for mapping modeling.

[0053] The present invention cleverly uses laser light instead of laser as the light source for data collection. The imaging of the laser light is as follows: Figure 3 As shown:

[0054] The advantages of this method are:

[0055] 1) The red and white spots of laser light imaging are prominent, and the pixel position can be easily extracted;

[0056] 2) Laser light is harmless to human body;

[0057] 3) The mapping parameters obtained using laser light as the light source are also applicable to the scene using laser as the energy source. In the two scenes, the galvanometer, camera and light source path have not changed.

[0058] The data collection process is completed automatically. The specific steps are as follows:

[0059] 1) Randomly set the galvanometer offset gm_dist(x,y) to move the laser point;

[0060] 2) Capture the image containing the laser point at this time through the camera;

[0061] 3) Use the template matching algorithm to obtain the position p(x,y) of the laser point on the image;

[0062] 4) Repeat the above steps 1) to 3) until n sets of corresponding data are obtained.

[0063] 3 Mapping Modeling

[0064] 3.1 Mapping from pixel coordinate system to galvanometer coordinate system

[0065] The simple pixel coordinate p(x,y) is mapped to the galvanometer offset gm_dist(x,y)', which lacks depth information. However, the depth will affect the galvanometer offset gm_dist(x,y)'. Therefore, it is necessary to:

[0066] Convert the target from the pixel coordinate system coordinates p(x,y) to the camera coordinate system coordinates c(x,y,z), that is:

[0067] c(x,y,z)=F p→c (p(x,y)) (2)

[0068] in:

[0069] F p→c It is the mapping function from pixel coordinates to camera coordinates, which can usually be obtained through the interface provided by the depth camera; it can also be obtained through camera calibration. This function is an inherent property of the camera;

[0070] z represents the depth distance of the target from the camera.

[0071] Convert the target coordinates c(x,y,z) in the camera coordinate system to g(x,y,z) in the galvanometer coordinate system, that is:

[0072] g(x,y,z)=F c→g (c(x,y,z)) (3)

[0073] The above transformation is due to the offset between the midpoint of the galvanometer coordinate system and the midpoint of the camera coordinate system, such as Figure 1 The blue and green coordinate systems are shown. Therefore, its specific form is as follows:

[0074]

[0075] in:

[0076] X, Y, and Z reflect the deviation between the midpoint of the camera coordinate system and the midpoint of the galvanometer coordinate system in the physical coordinate system. This deviation can be measured by a measuring tool. However, since the midpoint position of the galvanometer is difficult to determine, it can also be calculated using the least squares method through the collected data. The calculation process will be introduced in the remaining content.

[0077] 3.2 Pinhole Camera Model

[0078] After obtaining the coordinates of the target in the galvanometer coordinate system g(x,y,z), the next step is to complete the mapping of g(x,y,z) to the galvanometer offset gm(x,y) g→gm ,Right now:

[0079] gm(x,y)=F g→gm (g(x,y)) (4)

[0080] The mapping process can be modeled using a pinhole camera model, so the above formula becomes:

[0081]

[0082] The equivalent form is:

[0083]

[0084] in:

[0085] ·f x , f y is the scale factor

[0086] ·G x , G y It is the deviation between the midpoint of the galvanometer offset obtained by the pinhole camera model and the accurate midpoint of the galvanometer offset, which is similar to the offset between the pixel coordinates and the image coordinates in a pinhole camera. This item is equivalent to the buffer part and can effectively improve the overall mapping accuracy.

[0087] Note: An assumption here is that the x-direction and y-direction of the galvanometer offset gm(x,y) are independent, that is, they will not affect each other, so the matrix The value of the first row and second column, and the value of the second row and first column can be 0. However, this assumption is not accurate. Through the analysis of the light propagation path, it is known that the rotation of the galvanometer in two directions of the galvanometer scanner will affect the x-axis and y-axis of the light at the same time, so the simple pinhole camera model is not very accurate. The present invention introduces nonlinearity in the next 3.3 to improve the accuracy of the pinhole camera model.

[0088] 3.3 Nonlinear Model

[0089] According to the pinhole camera model in 3.2, the galvanometer offset gm x and gm y And the galvanometer coordinate system and There is a linear relationship between them. However, this is not the case in reality. The galvanometer offset gm x and gm y and and There is nonlinearity between them, and this nonlinearity is more obvious at the edge of the galvanometer offset, such as Figure 4 and Figure 5 As shown: Figure 4 is the galvanometer offset gm x and gm y Sampling point distribution map, Figure 5 is the galvanometer coordinate system and Sampling point distribution map.

[0090] It can be seen that: the galvanometer offset gm x and gm y The sampling points form a rectangle, and the corresponding and The sampling points are not a rectangle, and the straight sides of the rectangle become curved lines, which means that the data points are not linear. The nonlinearity of the galvanometer offset is also reflected in some papers on the properties of galvanometer scanners, such as Figure 6 As shown:

[0091] And the nonlinearity in the x-axis and y-axis directions is different and needs to be considered separately. For the x-axis direction, the nonlinearity increases with the increase of the x-axis, while the influence of the y-axis can be ignored. For the y-axis direction, it is similar to a radial distortion, which is affected by the combined influence of the x-axis and the y-axis.

[0092] 3.3.1 x-axis polynomial distortion modeling

[0093] Add distortion to the x-axis because it mainly depends on its own value. Therefore, a polynomial fit is used to model it. The formula is as follows:

[0094]

[0095] By balancing accuracy and computational complexity, it is recommended to set k to 3. The function form is as follows Figure 7 As shown:

[0096] 3.3.2 Modeling of y-axis radial distortion

[0097] For the y-axis, it uses the radial distortion model:

[0098] rr=(gm x -G x ) 2 +(gm y -G y ) 2 (6.1)

[0099]

[0100] 3.4 Formula Summary

[0101] At this point, the modeling of the entire model has been completed. Considering the distortion of the galvanometer offset, the mapping formula is:

[0102] gm_dist(x,y)'=F p→gm_dist (p(x,y)) (1)

[0103] A more specific calculation formula is as follows:

[0104] Convert the target from pixel coordinates to camera coordinates:

[0105] c(x,y,z)=F p→c (p(x,y)) (2)

[0106] Convert the target from the camera coordinate system to the galvo coordinate system:

[0107]

[0108] Convert the target from the galvo coordinate system to the galvo offset coordinate system:

[0109]

[0110] The galvanometer offset adds distortion in the x-axis and y-axis:

[0111]

[0112] rr=(gm x -G x ) 2 +(gm y -G y ) 2 (6.1)

[0113]

[0114] The final galvanometer offset is (gm_dist x ',gm_dist y '). The parameters to be determined during the entire mapping process include:

[0115] X, Y, Z: Deviation between the midpoint of the camera coordinate system and the midpoint of the galvanometer coordinate system in the physical coordinate system

[0116] f x , f y : Pinhole camera model scale factor

[0117] G x , G y : The deviation between the midpoint of the galvanometer offset obtained by the pinhole camera model and the accurate midpoint of the galvanometer offset is similar to the deviation between the pixel coordinates and the image coordinates in the pinhole camera.

[0118] X i : The distortion polynomial coefficient of the x-axis. In the case of a cubic polynomial fit, there are a total of X0 ~X 3 4 parameters

[0119] Y 1 : Y-axis radial distortion parameter

[0120] In summary, the overall mapping equation has 12 parameters to be determined.

[0121] 4 Parameter estimation

[0122] In Section 3, the mapping model has been built. Combined with the data collected in Section 2, the least squares method can be used to obtain parameters.

[0123] In this process, the input is the target camera coordinate system coordinates (c x ,c y ,c z ) and galvanometer offset (gm_dist x ,gm_dist y ).

[0124] in:

[0125] (c x ,c y ,c z ) The accuracy depends on the pixel coordinate accuracy of the target pixel point obtained by template matching and the corresponding conversion accuracy. Due to the good pixel characteristics of the laser point, the template matching method can obtain relatively accurate target pixel coordinates; while the conversion accuracy depends on the depth camera itself;

[0126] (gm_dist x ,gm_dist y ) is a random known value;

[0127] Through the above modeling, we can x ,c y ,c z ) is calculated (gm_dist x ',gm_dist y '). By minimizing (gm_dist x ,gm_dist y ) and (gm_dist x ',gm_dist y ') to obtain the parameters.

[0128] In summary, the above process can be summarized as follows:

[0129] The present invention combines a variety of technologies, including template matching, pinhole camera model, polynomial fitting, radial distortion model, least squares method and other technologies, to accurately map the target from the pixel value p(x, y) in the pixel coordinate system to the galvanometer offset gm_dist(x, y).

[0130] The sample data p(x,y) and gm_dist(x,y) are obtained by using laser light as the light source and combining it with template matching to obtain N groups (p(x,y), gm_dist(x,y)).

[0131] Next, we perform some coordinate system transformations on p(x,y) to obtain gm_dist(x,y)'. The specific steps are as follows:

[0132] 1) Using the depth camera parameters, transform the pixel coordinate system p(x,y) to the camera coordinate system c(x,y,z):

[0133] c(x,y,z)=F p→c (p(x,y)) (2)

[0134] 2) Convert the target from the camera coordinate system c(x,y,z) to the galvanometer scanner coordinate system g(x,y,z):

[0135]

[0136] 3) Convert the target from the galvanometer coordinate system g(x,y,z) to the galvanometer offset gm(x,y):

[0137]

[0138] 4) Apply distortion to the galvanometer offset gm(x,y) to obtain the final distortion offset gm_dist(x,y)':

[0139]

[0140] rr=(gm x -G x ) 2 +(gm y -G y ) 2 (6.1)

[0141]

[0142] 5) Construct the loss function between gm_dist(x,y) and gm_dist(x,y)', and obtain all parameters in the formula according to the least squares method;

[0143] 6) Save the calculated parameters X, Y, Z, f x、f y , G x , G y , X 0 ~X n , Y 1 , for subsequent use.

[0144] It should be noted that the above process can be fully automatically implemented, which means that the parameters can be updated at any time.

[0145] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0146] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for mapping pixel coordinates and galvanometer scanner offset based on laser point detection, characterized in that: By performing a series of coordinate system transformations on p(x,y), we get gm_dist(x,y)'. Among them, p(x,y) is the pixel coordinate system, gm_dist(x,y)' is the distortion offset calculated by the formula, The specific steps are: S100, using the depth camera parameters, convert the pixel coordinate system p(x,y) to the camera coordinate system c(x,y,z): c(x,y,z)=F p→c (p(x,y)), Among them, F p→c is the mapping function from pixel coordinates to camera coordinate system, z represents the depth distance of the target from the camera, S200, convert the target from the camera coordinate system c(x,y,z) to the galvanometer scanner coordinate system g(x,y,z): Among them, X, Y, and Z are the deviations between the midpoint of the camera coordinate system and the midpoint of the galvanometer coordinate system in the physical coordinate system. S300, convert the target from the galvanometer coordinate system g(x,y,z) into the galvanometer offset gm(x,y): Among them, f x , f y is the scale factor, G x , G y is the deviation between the midpoint of the galvanometer offset obtained by the pinhole camera model and the accurate midpoint of the galvanometer offset. S400, applying distortion to the galvanometer offset gm(x,y) to obtain the final distortion offset gm_dist(x,y)': rr=(gm x -G x ) 2 +(gm y -G y ) 2 Among them, X i : Distortion polynomial coefficient of the x-axis, Y1: radial distortion parameter of the y-axis, S500, construct a loss function between gm_dist(x,y) and gm_dist(x,y)', and obtain all parameters in the formula according to the least square method, where gm_dist(x,y) is the actual distortion offset, and gm_dist(x,y)' is the distortion offset calculated by the formula; S600, set the parameters X, Y, Z, f x 、f y , G x , G y 、X0~X n And Y1 is substituted into the above formulas S100-S400, and gm_dist(x,y)' is obtained according to p(x,y).

2. The method of pixel coordinate and galvanometer scanner offset mapping based on laser point detection according to claim 1, characterized in that: In the above-mentioned S400, the distortion polynomial coefficients of the x-axis are fitted by a third-order polynomial.

3. The method of pixel coordinate and galvanometer scanner offset mapping based on laser point detection according to claim 1, characterized in that: The acquisition of the p(x, y) and gm_dist(x, y) sample data adopts a laser lamp as a light source and combines it with a template matching method to obtain N groups (p(x, y), gm_dist(x, y)).

4. An application of a method for mapping pixel coordinates and galvanometer scanner offsets based on laser point detection, running a method for mapping pixel coordinates and galvanometer scanner offsets based on laser point detection as described in any one of claims 1 to 3, for any scenario in which the laser is controlled to operate through camera data.

5. A laser weeding robot, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the pixel coordinate and galvanometer scanner offset mapping method based on laser point detection as described in any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for mapping pixel coordinates and galvanometer scanner offset based on laser point detection according to any one of claims 1 to 3 is implemented.

Images