A laser weeding method based on vision guidance

Abstract

A laser weeding method based on vision guidance relates to the field of laser weeding technology. Existing laser weeding does not target meristems, but only weeds the location of weeds, resulting in low weeding accuracy and poor results. This application can identify weeds and the center point of the weed meristem, and send the location of the center point of the weed meristem to a laser device to ensure accurate irradiation of the weed meristem, providing a practical and precise weeding solution for laser weeding robots. This reduces damage to crops and improves weeding effectiveness. This application can cover large areas of farmland in a short period of time, processing several hectares of land per hour, and significantly improving weeding efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of laser weeding, in particular to a laser weeding method based on vision guidance. Background Art

[0002] Agricultural production is an indispensable component of a country's economic development. However, weeds continue to plague farmers. Weeds compete with crops for light, water, nutrients, and space, affecting crop growth and yield. Furthermore, weeds serve as hosts for pathogens and pests, spreading them and releasing chemicals that inhibit crop growth. Therefore, agricultural production requires more efficient and sustainable methods. Current traditional weed control methods primarily include chemical weeding, mechanical weeding, and manual weeding. While these methods are effective to a certain extent, they all have drawbacks.

[0003] Laser technology has been widely used in various fields (such as medicine, industrial processing, and communications). It has the advantages of high energy density, high precision, and non-contact processing. These advantages also make it promising in the field of weed control. Laser weed control robots can avoid damaging crops and polluting the environment, and can effectively reduce the use of pesticides. This technology not only helps to reduce agricultural production costs, but also improves the quality and safety of agricultural products. The effectiveness of laser weed control depends on the location of the weeds irradiated by the laser. Because the light intensity distribution of the laser beam usually exhibits a spherically symmetrical distribution with a stronger center and weaker surroundings, this distribution is called the Gaussian beam shape of the laser beam. Only when the laser irradiates the meristem of the weeds can a good weed control effect be achieved. Existing laser weed control does not target the meristem, but only the location of the weeds, resulting in low weed control accuracy and poor results. Summary of the Invention

[0004] The purpose of the present invention is to propose a laser weeding method based on vision guidance to address the problem that existing laser weeding does not target the meristem but only targets the weed position, resulting in low weeding accuracy and poor effect.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] A laser weeding method based on vision guidance, wherein the weeding method is implemented based on a laser weeding robot, wherein the laser weeding robot comprises a front camera 3, a coaxial camera 1, and a laser galvanometer 2;

[0007] The method comprises the following steps:

[0008] Step 1: Calibrate the front camera 3 and the laser galvanometer 2 to obtain the coordinate transformation matrix between the front camera 3 and the laser galvanometer 2;

[0009] Step 2: Calibrate the coaxial camera 1 and the laser galvanometer 2 to obtain the coordinate transformation matrix between the coaxial camera 1 and the laser galvanometer 2;

[0010] Step 3: Use the front camera 3 to acquire video data, and input each frame of the video data into the pre-trained first neural network to obtain the output prediction box and confidence score of the weed. Then, the weed prediction box with a confidence score higher than a threshold is selected as the weed target;

[0011] Step 4: Based on the weed targets, a multi-target tracking algorithm is used to track the weed targets in the video data. Based on the detection frame of each weed target, the center point coordinates of the detection frame are determined. Then, the center point coordinates corresponding to the weed targets are sorted from large to small according to the y-axis coordinates, and the sorting results are used as the processing queue;

[0012] Step 5: Select the weed targets in sequence according to the order in the processing queue, and obtain the center point coordinates corresponding to the weed targets. Then, use the coordinate transformation matrix between the front camera 3 and the laser galvanometer 2 to obtain the corresponding coordinates of the center point in the laser galvanometer 2 coordinate system, and deflect the laser galvanometer 2 to the coordinates;

[0013] Step 6: After the laser galvanometer 2 is deflected, it triggers the coaxial camera 1 to acquire the image within the field of view;

[0014] Step 7: Crop the weed targets in the detection frame to obtain a cropped image;

[0015] Step 8: Based on the image acquired by the coaxial camera 1, the image and the cropped image are input into a pre-trained second neural network to obtain the output coordinates of the meristem target point position;

[0016] Step 9: Using the coordinate transformation matrix between the coaxial camera 1 and the laser galvanometer 2, the position of the meristem target point is converted to the corresponding coordinates in the coordinate system of the laser galvanometer 2, the laser galvanometer 2 is deflected to the coordinates, and the laser is emitted;

[0017] Step 10: Repeat steps 5 to 9 until all weed targets in the processing queue are selected and laser weeding is completed.

[0018] Furthermore, the pre-trained first neural network is obtained by the following steps:

[0019] A field image containing weeds is obtained, and crops and weeds in the field image are labeled separately to construct training data, and the training data is used to train the first neural network.

[0020] Furthermore, the specific steps of step three are: using the front camera 3 to obtain video data, and inputting each frame image in the video data into a pre-trained first neural network to obtain the output prediction box and confidence of the crop, as well as the prediction box and confidence of the weed, and then selecting the weed prediction box with a confidence higher than the threshold as the weed target.

[0021] Furthermore, the step of tracking the weed targets in the video data using the multi-target tracking algorithm in step 4 includes:

[0022] Step 41: Select a frame from the video data as the current frame. Then, use the uniform motion model to estimate the position and size of the weed target in the previous frame in the current frame. Then, match the predicted weed target box detected in the current frame with the weed target in the previous frame. The matching is performed using the Intersection over Union (IoU) metric.

[0023] Step 42: Input the successfully matched target into the Kalman filter to obtain the optimal state estimation of the weed target. The optimal state estimation of the target includes the position and speed of the weed target. Finally, the speed of the weed target is used as the speed parameter of the uniform motion model of the next frame.

[0024] Furthermore, the pre-trained second neural network is obtained by the following steps:

[0025] The key points of the meristem in the weed target are marked, and then the cropped image and the normalized image are used as input, and the marked key points of the meristem are used as output to train the second neural network.

[0026] Furthermore, the second neural network specifically performs the following steps:

[0027] The second neural network extracts features from the input cropped image and the image from the coaxial camera 1, obtaining a weed feature vector of size N×1 and a feature vector of size M×1 containing weed features and meristem key point features. The weed feature vector and the feature vector containing weed features and meristem key point features are concatenated as the final features. The final features are passed through a multi-layer perceptron and regressed to obtain the key points of the weed meristem.

[0028] Furthermore, the method further includes a position compensation step for the time delay between the time t0 when the front camera acquires the video data and the time t1 when the laser galvanometer deflection starts in step 5. The compensation is expressed as:

[0029]

[0030]

[0031]

[0032] Where dt1 is the time interval between t1 and t0, δx1 is the offset in the x direction during the time (t1-t0), and δy1 is the offset in the y direction during the time (t1-t0). They represent the partial derivatives of the grayscale of the pixel in the image along the x, y, and t directions respectively, and u and v are the velocity vectors of the optical flow along the X and Y axes respectively.

[0033] Furthermore, the method further includes a position compensation step for the time delay between the time t0 when the front camera acquires the video data and the time t2 when the coaxial camera is triggered in step 6. The compensation is expressed as:

[0034]

[0035]

[0036]

[0037] Where dt2 is the time interval between t2 and t0, δx2 is the offset in the x direction within the time (t2-t0), and δy2 is the offset in the y direction within the time (t2-t0).

[0038] Furthermore, the method further includes a position compensation step for the time delay between the coaxial camera triggering time t2 in step 6 and the laser emitting time t3 in step 9, and the compensation is expressed as:

[0039]

[0040]

[0041]

[0042] Wherein, dt3 is the time interval between t3 and t2, δx3 is the offset in the x direction within the time (t3-t2), and δy3 is the offset in the y direction within the time (t3-t2).

[0043] Furthermore, the first neural network and the second neural network are YOLO models, and the multi-target tracking algorithm is SORT.

[0044] The beneficial effects of the present invention are:

[0045] This application can identify weeds and the center point of their meristems, transmitting the location of the weed meristem center point to a laser device, ensuring precise irradiation of the weed meristems and providing a practical and precise weed control solution for laser weed control robots. This reduces damage to crops and improves weed control effectiveness. This application can cover large areas of farmland in a short period of time, processing several hectares of land per hour, significantly improving weed control efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is the hardware configuration diagram of the control device for this application;

[0047] Figure 2 The overall process of vision-guided laser weed control technology for this application;

[0048] Figure 3 This is a schematic diagram of the meristem detection network model for this application;

[0049] Figure 4 This is the network structure diagram of the ResNet model in the meristem identification network of this application;

[0050] Figure 5 Schematic diagram of weed identification and meristem point detection results for this application. DETAILED DESCRIPTION

[0051] It should be noted that, unless there is any conflict, the various embodiments disclosed in this application can be combined with each other.

[0052] Specific implementation method 1: refer to Figure 1 Specifically describing this embodiment, a laser weeding method based on vision guidance is described in this embodiment. The weeding method is implemented based on a laser weeding robot. The laser weeding robot includes a front camera, a coaxial camera, and a laser galvanometer.

[0053] The method comprises the following steps:

[0054] Step 1: Calibrate the front camera and laser galvanometer to obtain the coordinate transformation matrix between the front camera and the laser galvanometer;

[0055] Step 2: Calibrate the coaxial camera and the laser galvanometer to obtain the coordinate transformation matrix between the coaxial camera and the laser galvanometer;

[0056] Step 3: Use the front camera to acquire video data and input each frame of the video data into the pre-trained first neural network to obtain the output weed prediction box and confidence score. Then, the weed prediction box with a confidence score higher than the threshold is selected as the weed target;

[0057] Step 4: Based on the weed targets, a multi-target tracking algorithm is used to track the weed targets in the video data. Based on the detection frame of each weed target, the center point coordinates of the detection frame are determined. Then, the center point coordinates are sorted from large to small based on the y-axis coordinates, and the sorting results are used as the processing queue;

[0058] Step 5: Select weed targets in sequence according to the order in the processing queue and obtain the center coordinates of the weed targets. Then, use the coordinate transformation matrix between the front camera and the laser galvanometer to obtain the corresponding coordinates of the center coordinates in the laser galvanometer coordinate system, and deflect the laser galvanometer to the coordinates.

[0059] Step 6: After the laser galvanometer is deflected, it triggers the coaxial camera to acquire the image within the field of view;

[0060] Step 7: Crop the weed targets in the detection frame to obtain a cropped image;

[0061] Step 8: Based on the image acquired by the coaxial camera, the image and the cropped image are input into a pre-trained second neural network to obtain the output coordinates of the meristem target point position;

[0062] Step nine: Use the coordinate transformation matrix between the coaxial camera and the laser galvanometer to convert the position of the meristem target point to the corresponding coordinate in the laser galvanometer coordinate system, deflect the laser galvanometer to the coordinate, and emit a laser (when this application is implemented, only the laser galvanometer is deflected, and the coaxial camera and the front camera are not deflected). This application uses the image obtained by the coaxial camera and the front camera image for comprehensive application to ensure the accuracy of meristem identification.

[0063] Step 10: Repeat steps 3 to 9 to complete laser weeding.

[0064] The hardware structure diagram of this application is as follows Figure 1 As shown, it includes a coaxial camera 1 for fine detection, a laser galvanometer 2 and a front camera 3 for rough detection.

[0065] like Figure 2 As shown in the figure, the implementation steps of vision-guided laser weeding technology are as follows. Take soybean field laser weeding as an example:

[0066] Step 1: Collect weed data and pre-train the model

[0067] Step 1.1 Use a data collection vehicle to capture soybean and weed data at 500 mm above the ground under different lighting conditions at random times in the morning, noon, and evening every day.

[0068] Step 1.2: Filter out images that contain weeds and are clear. Use labelme to annotate the weeds in these images and save the annotations in the formats specified by YOLO and ResNet. Then, divide the annotated information into training, validation, and test sets.

[0069] In step 1.3, perform data augmentation on the images (such as random cropping, rotation, flipping, and adjusting brightness and contrast) to increase data diversity and improve the model's generalization capabilities. The model is then trained on the processed training dataset. Transfer learning is then performed on the ResNet model, using the parameters of the pretrained model as the initial parameters and training on the target task data to adapt the model to the target task.

[0070] Step 1.4 exports the trained model weights into a format suitable for deployment, deploys the model in the actual application environment, and optimizes the inference speed.

[0071] Step 2: Deploy the trained model for weed detection

[0072] Step 2.1 calibrates the front camera and coaxial camera to determine the internal and external parameters of the cameras so as to accurately map the information in the 2D image to the 3D world.

[0073] In step 2.2, while the laser weeding robot is operating, the trigger thread for the front camera is started, causing it to send signals at a set interval. Each time a trigger signal is sent, an image within the 600mm x 450mm field of view of the front camera is captured and transmitted to the coarse weed detection thread for processing.

[0074] In step 2.3, resize the input image to the required dimensions and perform normalization and tensor conversion. The preprocessed image is then fed into the YOLO model. The model extracts weed image features through a series of convolutional layers and outputs predictions at multiple scales. The anchor box offsets output by the model are converted to bounding box coordinates in the actual image. Based on the prediction confidence, low-confidence predictions are filtered out. The class probabilities of each predicted box are filtered, retaining only those with probabilities above a certain threshold. This process filters out weeds within the front camera.

[0075] Step 3: Use SORT algorithm for multi-target tracking

[0076] Step 3.1 In each frame, the SORT multi-target tracking algorithm uses YOLO to detect the target in the image and obtain the target's bounding box and category confidence.

[0077] Step 3.2 applies the Kalman filter to each tracked target and uses the position and velocity information of the previous frame to predict the position and obtain the predicted position of each target.

[0078] In step 3.3, the cost matrix is ​​constructed by calculating the IOU between the detection box and the prediction box, and the best match between the detection box and the tracking box is found to complete the target association and assign a unique ID number in the continuous frame to each weed. The target appearing in the current image frame is loaded into the task planning thread in the form of ID number + detection box.

[0079] Each weeding task thread submits its own weeding task request to the task planning thread and stores all target tracking results obtained from the detection program. The target's location and appearance image information are quickly indexed by ID number. The ID number of each weed target detected in each frame is used to index the corresponding element in the memory and assign and update the value. The task request is then processed one by one, and information such as the location, appearance image, and detection timestamp of any weed target that has entered the laser's operating range is transmitted. After the transmission is completed, the target information is erased.

[0080] Multi-target tracking using Sort algorithm

[0081] In each frame, the SORT multi-object tracking algorithm uses YOLO to detect the target in the image and obtain the target's bounding box and category confidence.

[0082] The uniform motion model is used to estimate the position and size of the target in the previous frame in the current frame. The target detected by YOLO in the current frame is then matched with the target in the previous frame, usually using the Intersection over Union (IoU) metric. Matched targets represent the same object, while unmatched targets may be newly appeared or disappeared.

[0083] The Kalman filter generates an optimal state estimate for the matched target. The Kalman filter combines the observations from the current frame with the motion model estimation from the previous frame to arrive at an optimal estimate of the target's state. This includes information such as position and velocity. This completes target association, assigning each weed a unique ID within consecutive frames, and loading the target appearing in the current image frame into the task planning thread as an ID number and a detection box.

[0084] Step 4: Take the task to deflect the galvanometer and use the optical flow field to compensate for the displacement

[0085] Optical flow compensation principle:

[0086] The front camera is triggered rapidly and continuously to acquire continuous frame images, and the timestamp of each frame image acquisition is recorded. The feature point extraction algorithm is used to extract high-quality feature points in the image, and the feature points are located and tracked on two consecutive frames. The pixel position vector of the feature point on the two consecutive frames is converted into a position vector in the actual physical coordinate system. The corresponding timestamp of the image recorded in 1 is then used to calculate the time difference between the two frames to determine the optical flow speed at the current moment. The sliding window filtering algorithm is applied to eliminate the impact of instantaneous speed mutations to improve the stability of the algorithm in obtaining speed. The processing time of the camera's continuous frame recognition of weeds and meristem key points is calculated. At the current moment, the speed and processing time calculated by the optical flow field are used to compensate for the positioning delay on the coordinates detected by the camera application model.

[0087] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solutions of the present invention and cannot be used to limit the scope of protection. Any minor changes made based on the claims and description of the present invention shall still fall within the scope of protection of the present invention.

Claims

1. A laser weeding method based on vision guidance, characterized in that The weeding method is implemented based on a laser weeding robot, which comprises a front camera (3), a coaxial camera (1) and a laser galvanometer (2); The method comprises the following steps: Step 1: Calibrate the front camera (3) and the laser galvanometer (2) to obtain a coordinate transformation matrix between the front camera (3) and the laser galvanometer (2); Step 2: Calibrate the coaxial camera (1) and the laser galvanometer (2) to obtain a coordinate transformation matrix between the coaxial camera (1) and the laser galvanometer (2); Step 3: Use the front camera (3) to acquire video data, and input each frame of the video data into the pre-trained first neural network to obtain the output prediction box and confidence of the weeds, and then select the weed prediction box with a confidence higher than a threshold as the weed target; Step 4: Based on the weed targets, a multi-target tracking algorithm is used to track the weed targets in the video data. Based on the detection frame of each weed target, the center point coordinates of the detection frame are determined. Then, the center point coordinates corresponding to the weed targets are sorted from large to small according to the y-axis coordinates, and the sorting results are used as the processing queue; Step 5: Select the weed targets in sequence according to the order in the processing queue, and obtain the center point coordinates corresponding to the weed targets. Then, use the coordinate transformation matrix between the front camera (3) and the laser galvanometer (2) to obtain the coordinates corresponding to the center point coordinates in the laser galvanometer (2) coordinate system, and deflect the laser galvanometer (2) to the coordinates. Step 6: After the laser galvanometer (2) is deflected, it triggers the coaxial camera (1) to acquire the image within the field of view; Step 7: Crop the weed targets in the detection frame to obtain a cropped image; Step 8: Based on the image acquired by the coaxial camera (1), the image and the cropped image are input into a pre-trained second neural network to obtain the output coordinates of the meristem target point position; Step 9: Using the coordinate transformation matrix between the coaxial camera (1) and the laser galvanometer (2), the position of the meristem target point is converted to the corresponding coordinates in the laser galvanometer (2) coordinate system, the laser galvanometer (2) is deflected to the coordinates, and the laser is emitted; Step 10: Repeat steps 5 to 9 until all weed targets in the processing queue are selected and laser weeding is completed; The first neural network and the second neural network are YOLO models, and the multi-target tracking algorithm is SORT.

2. A laser weeding method based on vision guidance according to claim 1, characterized in that The pre-trained first neural network is obtained by the following steps: A field image containing weeds is obtained, and crops and weeds in the field image are labeled separately to construct training data, and the training data is used to train the first neural network.

3. The laser weeding method based on vision guidance according to claim 1, characterized in that The step of tracking the weed targets in the video data using the multi-target tracking algorithm in step 4 includes: Step 41: Select a frame from the video data as the current frame. Then, use the uniform motion model to estimate the position and size of the weed target in the previous frame in the current frame. Then, match the predicted weed target box detected in the current frame with the weed target in the previous frame. The matching is performed using the Intersection over Union (IoU) metric. Step 42: Input the successfully matched target into the Kalman filter to obtain the optimal state estimation of the weed target. The optimal state estimation of the target includes the position and speed of the weed target. Finally, the speed of the weed target is used as the speed parameter of the uniform motion model of the next frame.

4. The laser weeding method based on vision guidance according to claim 1, characterized in that The pre-trained second neural network is obtained by the following steps: The key points of the meristem in the weed target are marked, and then the cropped image and the normalized image are used as input, and the marked key points of the meristem are used as output to train the second neural network.

5. The laser weeding method based on vision guidance according to claim 4, characterized in that The second neural network specifically performs the following steps: The second neural network extracts features from the input cropped image and the coaxial camera (1) image respectively, and obtains a weed feature vector of size N×1 and a feature vector of size M×1 containing weed features and meristem key point features. The weed feature vector and the feature vector containing weed features and meristem key point features are concatenated as the final features. The final features are passed through a multi-layer perceptron and regressed to obtain the key points of the weed meristem.

6. The laser weeding method based on vision guidance according to claim 1, characterized in that The method further includes a position compensation step for the time delay between the time t0 when the front camera acquires the video data and the time t1 when the laser galvanometer deflection starts in step 5. The compensation is expressed as: Where dt1 is the time interval between t1 and t0, δx1 is the offset in the x direction during the time between t1 and t0, and δy1 is the offset in the y direction during the time between t1 and t0. They represent the partial derivatives of the grayscale of the pixel in the image along the x, y, and t directions respectively, and u and v are the velocity vectors of the optical flow along the X and Y axes respectively.

7. The laser weeding method based on vision guidance according to claim 6, characterized in that The method further includes a position compensation step for the time delay between the time t0 when the front camera acquires the video data and the time t2 when the coaxial camera is triggered in step 6. The compensation is expressed as: Where dt2 is the time interval between t2 and t0, δx2 is the offset in the x direction during the time t2-t0, and δy2 is the offset in the y direction during the time t2-t0.

8. The laser weeding method based on vision guidance according to claim 7, characterized in that The method further includes a position compensation step for the time delay between the coaxial camera triggering time t2 in step 6 and the laser emitting time t3 in step 9, the compensation being expressed as: Wherein, dt3 is the time interval between t3 and t2, δx3 is the offset in the x direction during the time t3-t2, and δy3 is the offset in the y direction during the time t3-t2.

Images