Crop seedling positioning method

Abstract

The present invention belongs to the field of agricultural machinery automation technology and discloses a method for locating crop seedlings. The method includes: first, fixing the relative position of a camera and a laser radar; second, aligning the camera and the laser radar in the time dimension and the spatial dimension; third, improving a target detection model to obtain a recognition model of the crop seedlings; fourth, projecting the laser radar point cloud onto the camera image; fifth, performing a clustering operation on the point cloud in the image bounding box to obtain the center point coordinates of the point cloud; and sixth, outputting the spatial position coordinates of the crops. The present invention fully utilizes the advantages of cameras and laser radars, realizes the integration of the advantages of the two, and achieves the purpose of locating crop seedlings, which is impossible for existing cameras or laser radars to accomplish alone. The present invention can serve as an upstream technology for the research and development of intelligent farmland weeding machinery.

Description

Technical Field

[0001] The invention belongs to the technical field of agricultural machinery automation, and in particular relates to a crop seedling positioning method. Background Art

[0002] There are two main strategies for intelligent robotic weeding. One is to first locate weeds and then remove them at specific locations. The other is to accurately locate crop seedlings, then avoid them and uniformly perform weeding operations in areas without them. Compared to weeds, crops grow in more regular patterns, making it preferable to locate the seedlings and then perform weeding operations. This is especially true for shorter crops, such as lettuce, cabbage, and cauliflower. Simply determining the centroid of a single plant allows the weeding actuator to precisely avoid it. However, while camera-based recognition models trained using deep learning can currently identify crop seedlings with high accuracy, accurately acquiring their location information is difficult. LiDAR can accurately acquire their location information, but identifying the crop species is more challenging. To address these shortcomings, we have proposed a crop seedling location method that combines the advantages of cameras and LiDAR. Summary of the Invention

[0003] The present invention aims to provide a method for locating crop seedlings. The method employs a strategy of fusing camera images with lidar point clouds, and includes the following steps:

[0004] S1, fix the relative position of the camera and lidar;

[0005] S2. Align the camera and the lidar in the time dimension and space dimension; compare the camera's shooting frame rate with the lidar's scanning frequency, use the lower frequency as the benchmark, extract the data obtained by the higher frequency at intervals, set the data extraction frequency to the lower frequency of the two, and use the timestamp as the standard for aligning the camera and the lidar in the time dimension; perform homogeneous transformation on the camera coordinate system and the lidar coordinate system respectively, and then fuse the lidar coordinate system into the image coordinate system according to the following formula,

[0006] q i =K[RT]p i

[0007] Among them, q i represents the i-th radar data to be projected onto the image coordinate system, K represents the intrinsic parameter matrix of the camera, R represents the rotation matrix that fuses the radar coordinate system into the image coordinate system, T represents the translation matrix that fuses the radar coordinate system into the image coordinate system, and p i Represents the i-th radar data in the radar coordinate system;

[0008] S3. Obtain a crop seedling recognition model; improve a single shot detection (SSD) model, use a camera to capture images containing crop seedlings in a farmland, train a crop seedling recognition model, and then use a rectangular box to surround the identified crop seedlings;

[0009] S4, projecting the lidar point cloud onto the camera image;

[0010] S5. Perform a clustering operation on the point cloud in the image bounding box to obtain the coordinates of the center point of the point cloud. Specifically, based on the coordinates of the upper left corner and the lower right corner of the crop seedling bounding box identified in S3, obtain the vertical center line of the bounding box. With the vertical center line as the central axis, reduce the width of the bounding box to one-third of the original bounding box. Screen the radar point cloud in the small bounding box. When the radar point meets the following formula, the radar point will be used for clustering operation. Then, the average of the three-dimensional coordinate values ​​of all radar points used for clustering operation is calculated, and then a new spatial coordinate point is generated. This new spatial coordinate point is considered to be the centroid position of the crop seedling, and the centroid represents the spatial position of the crop seedling.

[0011] f(x)-R th ≤L dd ≤f(x)+R th

[0012] Where f(x) represents the average distance from the origin of the image coordinate system to the crop seedlings obtained according to the planting pattern and camera position, L dd Represents the detection distance from the new spatial coordinate point to the origin of the image coordinate system, R th Represents the set threshold;

[0013] S6. Output the spatial position coordinates of the crop seedlings.

[0014] Preferably, in step S1, the scanning center plane of the laser radar is adjusted to align with the center height of the camera image according to the scale of the camera image.

[0015] Preferably, step S3 retains the link between the last feature map and the output layer in the classic SSD model, and discards the link between the first five feature maps and the last output layer.

[0016] Preferably, for the image containing crop seedlings taken in step S3, a visual image calibration tool is used to calibrate the target crop seedlings in the image and generate corresponding labels, ignoring background objects in the image; this operation achieves the purpose of automatically filtering background objects for the crop seedling recognition model subsequently obtained through deep learning training.

[0017] Compared with the prior art, the present invention provides a method for positioning crop seedlings, which has the following beneficial effects:

[0018] 1. The present invention fully utilizes the advantages of cameras and lidars. With cameras as hardware, deep learning technology is used to obtain an identification model that can accurately identify crop seedlings. With lidars as hardware, the orientation information of crop seedlings can be accurately determined. By inventing a method for fusing cameras and lidars, the advantages of the two are integrated, achieving the purpose of positioning crop seedlings, which cannot be accomplished by existing cameras or lidars alone.

[0019] 2. In the classic SSD model, the larger feature maps at the front are suitable for identifying subtle features, while the smaller feature maps at the back are suitable for identifying macro features. Since the present invention is application-oriented, in actual operations, cameras will deliberately capture crops in farmland, so crops will be obvious objects with macro features in the image. Therefore, the present invention retains the link between the last feature map and the output layer in the classic SSD model, and discards the link between the first five feature maps and the final output layer. This invention can not only accurately identify crops in images, but also improves the algorithm's running speed.

[0020] 3. The present invention reduces the width of the bounding box to one-third of the original bounding box, which can more accurately lock the lidar point near the centroid of the crop seedling. Compared with not reducing the bounding box, the present invention has higher positioning accuracy for the crop seedling. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the present invention;

[0022] Figure 2 Schematic diagram of the network of the "Improved Single Shot Detection (SSD) Model" of the present invention;

[0023] Figure 3 Results of the "Improved One-Shot Detection (SSD) Model" tested in this invention for identifying lettuce;

[0024] Figure 4 The result of projecting the single-line laser radar point cloud tested by the present invention onto the camera image;

[0025] Figure 5 This is the new bounding box and lidar point cloud in the present invention that are reduced to one-third of the original bounding box in the width direction. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] Please see the attached Figure 1-5 , taking the positioning of lettuce (Lactuca sativa L.var.ramosa Hort.) seedlings in the farmland as an example, first align the scanning center plane of the lidar with the center height of the camera image according to the scale of the camera image, and then fix the camera and lidar on the frame of any agricultural machinery. A camera with a frame rate of 40fps (i.e. 40Hz) and a lidar with a scanning frequency of 10Hz are used. Based on the scanning frequency of the lidar, camera images are extracted at a frequency of every 4 images (i.e. every 3 images), and the timestamps of the camera and lidar are used to align the starting time of the two. Refer to the instruction manual or user manual of the camera to obtain the internal parameters of the camera. Perform homogeneous transformations on the camera coordinate system and the lidar coordinate system respectively, and then fuse the lidar coordinate system into the image coordinate system according to the following formula,

[0028] q i =K[RT]p i

[0029] Among them, q i represents the i-th radar data to be projected onto the image coordinate system, K represents the intrinsic parameter matrix of the camera, R represents the rotation matrix that fuses the radar coordinate system into the image coordinate system, T represents the translation matrix that fuses the radar coordinate system into the image coordinate system, and p i Represents the i-th radar data in the radar coordinate system.

[0030] Collect a large number of lettuce images in the farmland, use the LabelImg plug-in to mark the lettuce in the image, and generate the corresponding label. Keep the last feature map and the output layer link in the classic SSD model, and discard the first 5 feature maps and the last output layer link, such as Figure 2 As shown. Using Tensorflow or Baidu PaddlePaddle as the deep learning framework, the above lettuce image and corresponding labels are imported into the improved SSD model for training to obtain the lettuce recognition model. When the model is used to recognize lettuce again, a rectangular box will surround the lettuce in the image, as shown in the figure below. Figure 3 shown.

[0031] Project the lidar point cloud onto the camera image, such as Figure 4Perform clustering on the point cloud in the image bounding box to obtain the coordinates of the center point of the point cloud; specifically, based on the coordinates of the upper left corner and lower right corner of the crop seedling bounding box, obtain the vertical center line of the bounding box. With the vertical center line as the central axis, reduce the width of the bounding box to one-third of the original bounding box, as shown in Figure 1. Figure 5 As shown in the figure, the radar point cloud in the small bounding box is screened. When the radar point meets the following formula, the radar point will be used for clustering operation. Then the average value of the three-dimensional coordinate values ​​of all radar points used for clustering operation is calculated, and a new spatial coordinate point is generated. This new spatial coordinate point is considered to be the centroid position of the crop seedling, and the centroid represents the spatial position of the crop seedling.

[0032] f(x)-R th ≤L dd ≤f(x)+R th

[0033] Where f(x) represents the average distance from the origin of the image coordinate system to the crop seedlings obtained according to the planting pattern and camera position, L dd Represents the detection distance from the new spatial coordinate point to the origin of the image coordinate system, R th Represents the set threshold.

[0034] Finally, based on the image coordinate system (i.e., in the image coordinate system), the three-dimensional coordinate values ​​of the new spatial coordinate points are output. At this point, the specific orientation information of the lettuce seedlings relative to the camera is obtained, and the positioning of the lettuce seedlings is completed.

[0035] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for locating crop seedlings, using a strategy of fusing camera images with lidar point clouds, characterized by: The following steps are involved: S1, fix the relative position of the camera and lidar; S2. Compare the camera's shooting frame rate and the lidar's scanning frequency. Using the lower frequency as the benchmark, extract the data obtained from the higher frequency at intervals. Set the data extraction frequency to the lower frequency of the two. Use the timestamp as the standard for aligning the camera and lidar in the time dimension. Perform homogeneous transformations on the camera coordinate system and the lidar coordinate system respectively, and then fuse the lidar coordinate system into the image coordinate system according to the following formula: q i =K[RT]p i Among them, q i represents the i-th radar data to be projected onto the image coordinate system, K represents the intrinsic parameter matrix of the camera, R represents the rotation matrix that fuses the radar coordinate system into the image coordinate system, T represents the translation matrix that fuses the radar coordinate system into the image coordinate system, and p i Represents the i-th radar data in the radar coordinate system; S3. Improve the single shot detection (SSD) model by using a camera to capture images of crop seedlings in a farmland, train a crop seedling recognition model, and then surround the identified crop seedlings with a rectangular box. S4, projecting the lidar point cloud onto the camera image; S5. Based on the coordinates of the upper left corner and the lower right corner of the crop seedling bounding box identified in S3, obtain the vertical center line of the bounding box. Using the vertical center line as the central axis, reduce the width of the bounding box to one-third of the original bounding box. Filter the radar point cloud in the small bounding box. When a radar point meets the following formula, the radar point will be used for clustering operation. Then, the average of the three-dimensional coordinate values ​​of all radar points used for clustering operation is calculated, and a new spatial coordinate point is generated. f(x)-R th ≤L dd ≤f(x)+R th Where f(x) represents the average distance from the origin of the image coordinate system to the crop seedlings obtained according to the planting pattern and camera position, L dd Represents the detection distance from the new spatial coordinate point to the origin of the image coordinate system, R th Represents the set threshold; S6. Output the spatial position coordinates of the crop seedlings.

2. A crop seedling positioning method according to claim 1, characterized in that: In step S1, the scanning center plane of the laser radar is adjusted to align with the center height of the camera image according to the scale of the camera image.

3. A method for positioning crop seedlings according to claim 1, characterized in that: Step S3 retains the link between the last feature map and the output layer in the classic SSD model, and discards the link between the first five feature maps and the last output layer.

4. A method for positioning crop seedlings according to claim 1, characterized in that: For the image containing the crop seedlings captured in step S3, a visual image calibration tool is used to calibrate the target crop seedlings in the image and generate corresponding labels, ignoring background objects in the image.

Images