Laser weeding method based on visual guidance

Through visually guided laser weeding method, cameras and neural networks are used to identify and track weed meristems to ensure accurate deflection of laser galvanometers, solving the problem of low weeding accuracy in the existing technology, and achieving efficient weeding effect .

CN119007148AActive Publication Date: 2024-11-22HARBIN INST OF TECH

Patent Information

Application Number
CN202411039725.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-22
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

The existing laser herbicidal technology fails to accurately irradiate the meristems of weeds, resulting in low accuracy and poor effect.

Method used

Using a vision-guided laser weeding method, the front camera and coaxial camera combined with a pre-trained neural network are used to identify and track weeds and meristems, and the laser galvanometer is ensured accurately deflected by a multi-objective tracking algorithm and coordinate transformation matrix. to the location of meristem.

Benefits of technology

Accurate identification and irradiation of weed meristems is achieved, the accuracy and efficiency of weeding is improved, the damage to crops is reduced, and the weeding effect is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007148A_ABST
    Figure CN119007148A_ABST
Patent Text Reader

Abstract

The invention discloses a laser weeding method based on visual guidance, relates to the technical field of laser weeding, and aims to solve the problems of low weeding accuracy and poor weeding effect due to the fact that existing laser weeding is not carried out on meristem but only carried out on weed positions. The central point position of the meristem of the weed is sent to the laser device, it is ensured that the meristem of the weed is accurately irradiated, and an accurate weeding scheme capable of being practically applied is provided for the laser weeding robot. The damage to crops is reduced, and the weeding effect is improved. According to the application, a large area of farmland can be covered in a short time, hectare of land can be treated per hour, and the weeding efficiency is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of laser weeding, and in particular to a laser weeding method based on vision guidance. Background Art

[0002] Agricultural production is an indispensable part of a country's economic development. However, the problem of weeds still plagues farmers. Weeds compete with crops for light, water, nutrients and space, affecting crop growth and yield. In addition, weeds serve as hosts for pathogens and pests, spread pests and diseases, and inhibit crop growth by releasing chemicals. Therefore, agricultural production requires a more efficient and sustainable way. The current traditional weed control methods mainly include chemical weed control, mechanical weed control and manual weed control. Although these methods are effective to a certain extent, they all have some shortcomings.

[0003] Laser technology has been widely used in many fields (such as medical treatment, industrial processing, communications, etc.). It has the advantages of high energy density, high precision, and non-contact processing. These advantages make it also have potential in the field of weeding. Laser weeding robots can avoid damage to crops and pollution to the environment, and can effectively reduce the use of pesticides. This technology not only helps to reduce agricultural production costs, but also improves the quality and safety of agricultural products. The effect of laser weeding depends on the part of the weeds irradiated by the laser. Because the light intensity distribution of the laser beam usually presents a spherically symmetrical distribution form with a strong center and weak surroundings, this distribution form is called the Gaussian beam form of the laser beam. Only when the laser irradiates the meristem of the weeds can a good weeding effect be achieved. Existing laser weeding does not target the meristem, but only targets the weed position, resulting in low weeding accuracy and poor effect. Summary of the invention

[0004] The purpose of the present invention is to propose a laser weeding method based on vision guidance to solve the problem that the existing laser weeding does not target the meristem but only targets the weed position, thus resulting in low weeding accuracy and poor effect.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] A laser weeding method based on vision guidance, the weeding method is implemented based on a laser weeding robot, the laser weeding robot includes a front camera 3, a coaxial camera 1 and a laser galvanometer 2;

[0007] The method comprises the following steps:

[0008] Step 1: calibrate the front camera 3 and the laser galvanometer 2 to obtain a coordinate transformation matrix between the front camera 3 and the laser galvanometer 2;

[0009] Step 2: Calibrate the coaxial camera 1 and the laser galvanometer 2 to obtain a coordinate transformation matrix between the coaxial camera 1 and the laser galvanometer 2;

[0010] Step 3: Use the front camera 3 to obtain video data, and input each frame of the video data into the pre-trained first neural network to obtain the output prediction box and confidence of the weeds, and then select the weed prediction box with a confidence higher than a threshold as the weed target;

[0011] Step 4: Based on the weed targets, a multi-target tracking algorithm is used to track the weed targets in the video data, and based on the detection frame of each weed target, the center point coordinates of the detection frame are determined. Then, the y-axis coordinates of the center point coordinates corresponding to the weed targets are sorted from large to small, and the sorting results are used as a processing queue;

[0012] Step 5: Select the weed targets in sequence according to the order in the processing queue, and obtain the center point coordinates corresponding to the weed targets. Then, use the coordinate transformation matrix between the front camera 3 and the laser galvanometer 2 to obtain the coordinates corresponding to the center point coordinates in the laser galvanometer 2 coordinate system, and deflect the laser galvanometer 2 to the coordinates;

[0013] Step 6: After the laser galvanometer 2 is deflected, it triggers the coaxial camera 1 to acquire the image in the field of view;

[0014] Step 7: Crop the weed targets in the detection frame to obtain a cropped image;

[0015] Step 8: Based on the image acquired by the coaxial camera 1, the image and the cropped image are input into a pre-trained second neural network to obtain the output coordinates of the meristem target point position;

[0016] Step nine: using the coordinate transformation matrix between the coaxial camera 1 and the laser galvanometer 2, the position of the meristem target point is converted to the corresponding coordinates in the coordinate system of the laser galvanometer 2, so that the laser galvanometer 2 is deflected to the coordinates and emits laser light;

[0017] Step 10: Repeat steps 5 to 9 until all weed targets in the processing queue are selected and laser weeding is completed.

[0018] Furthermore, the pre-trained first neural network is obtained by the following steps:

[0019] A field image containing weeds is obtained, and crops and weeds in the field image are respectively labeled to construct training data, and the training data is used to train the first neural network.

[0020] Furthermore, the specific steps of step three are: using the front camera 3 to obtain video data, and inputting each frame image in the video data into a pre-trained first neural network to obtain the output prediction box and confidence of the crop, as well as the prediction box and confidence of the weeds, and then selecting the weed prediction box with a confidence higher than a threshold as the weed target.

[0021] Furthermore, the step of tracking the weed targets in the video data using the multi-target tracking algorithm in step 4 includes:

[0022] Step 41: Select a frame in the video data as the current frame, then use the uniform motion model to estimate the position and size of the weed target in the previous frame in the current frame, and then match the weed target prediction box detected in the current frame with the weed target in the previous frame. The matching is performed using the IoU metric;

[0023] Step 42: Input the successfully matched target into the Kalman filter to obtain the optimal state estimation of the weed target, wherein the optimal state estimation of the target includes the position and speed of the weed target, and finally the speed of the weed target is used as the speed parameter of the uniform motion model of the next frame.

[0024] Furthermore, the pre-trained second neural network is obtained by the following steps:

[0025] The key points of the meristem in the weed target are marked, and then the cropped image and the normalized image are used as input, and the marked key points of the meristem are used as output to train the second neural network.

[0026] Furthermore, the second neural network specifically performs the following steps:

[0027] The second neural network extracts features from the input cropped image and the image from the coaxial camera 1, respectively, to obtain a weed feature vector of size N×1 and a feature vector of size M×1 containing weed features and meristem key point features. The weed feature vector and the feature vector containing weed features and meristem key point features are concatenated as the final features. The final features are regressed through a multi-layer perceptron to obtain the key points of the weed meristem.

[0028] Furthermore, the method further includes the step of acquiring video data at a time t of the front camera. 0 The laser galvanometer deflection start time t in step 5 1 The position compensation step of the time delay between is expressed as:

[0029]

[0030]

[0031]

[0032] Among them, dt 1 t 1 With t 0 The time interval, δx 1 is(t 1 -t 0 ) time in the x direction, δy 1 is(t 1 -t 0 ) time in the y direction, They represent the partial derivatives of the grayscale of the pixel in the image along the x, y, and t directions respectively, and u and v are the velocity vectors of the optical flow along the X-axis and Y-axis respectively.

[0033] Furthermore, the method further includes the step of acquiring video data at a time t of the front camera. 0 The coaxial camera triggering time t in step 6 2 The position compensation step of the time delay between is expressed as:

[0034]

[0035]

[0036]

[0037] Among them, dt 2 t 2 With t 0 The time interval, δx 2 is(t 2 -t 0 ) time in the x direction, δy 2 is(t 2 -t 0 ) time in the y direction.

[0038] Furthermore, the method further includes triggering the coaxial camera at the time t 2 The laser emission time t in step 9 3 The position compensation step of the time delay between is expressed as:

[0039]

[0040]

[0041]

[0042] Among them, dt 3 t 3 With t 2The time interval, δx 3 is(t 3 -t 2 ) time in the x direction, δy 3 is(t 3 -t 2 ) time in the y direction.

[0043] Furthermore, the first neural network and the second neural network are YOLO models, and the multi-target tracking algorithm is SORT.

[0044] The beneficial effects of the present invention are:

[0045] This application can identify weeds and the center point of the weed meristem, and send the location of the center point of the weed meristem to the laser device to ensure accurate irradiation of the weed meristem, providing a practical and accurate weeding solution for the laser weeding robot. It reduces damage to crops and improves the weeding effect. This application can cover a large area of ​​farmland in a short period of time, can process several hectares of land per hour, and greatly improve the weeding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is the hardware structure diagram of the control device for this application;

[0047] Figure 2 The overall process of the vision-guided laser weeding technology for this application;

[0048] Figure 3 This is a schematic diagram of the meristem detection network model for this application;

[0049] Figure 4 This is the network structure diagram of the ResNet model in the meristem identification network of this application;

[0050] Figure 5 This is a schematic diagram of the weed identification and meristem point detection results for this application. DETAILED DESCRIPTION

[0051] It should be particularly noted that, in the absence of conflict, the various embodiments disclosed in this application can be combined with each other.

[0052] Specific implementation method 1: refer to Figure 1 Specifically describe this embodiment, a laser weeding method based on vision guidance described in this embodiment, the weeding method is implemented based on a laser weeding robot, and the laser weeding robot includes a front camera, a coaxial camera and a laser galvanometer;

[0053] The method comprises the following steps:

[0054] Step 1: Calibrate the front camera and the laser galvanometer to obtain the coordinate transformation matrix between the front camera and the laser galvanometer;

[0055] Step 2: Calibrate the coaxial camera and the laser galvanometer to obtain the coordinate transformation matrix between the coaxial camera and the laser galvanometer;

[0056] Step 3: Use the front camera to obtain video data, and input each frame of the video data into the pre-trained first neural network to obtain the output prediction box and confidence of the weeds, and then select the weed prediction box with a confidence higher than the threshold as the weed target;

[0057] Step 4: Based on the weed targets, a multi-target tracking algorithm is used to track the weed targets in the video data, and based on the detection frame of each weed target, the center point coordinates of the detection frame are determined. Then, the center point coordinates are sorted from large to small according to the y-axis coordinates, and the sorting results are used as a processing queue;

[0058] Step 5: Select the weed targets in sequence according to the order in the processing queue, and obtain the center point coordinates corresponding to the weed targets. Then, use the coordinate transformation matrix between the front camera and the laser galvanometer to obtain the coordinates corresponding to the center point coordinates in the laser galvanometer coordinate system, and deflect the laser galvanometer to the coordinates;

[0059] Step 6: After the laser galvanometer is deflected, the coaxial camera is triggered to acquire the image within the field of view;

[0060] Step 7: Crop the weed targets in the detection frame to obtain a cropped image;

[0061] Step 8: Based on the image acquired by the coaxial camera, the image and the cropped image are input into a pre-trained second neural network to obtain the output coordinates of the meristem target point position;

[0062] Step nine: Use the coordinate transformation matrix between the coaxial camera and the laser galvanometer to convert the position of the meristem target point to the corresponding coordinates in the laser galvanometer coordinate system, deflect the laser galvanometer to the coordinates, and emit a laser (when this application is implemented, only the laser galvanometer is deflected, and the coaxial camera and the front camera are not deflected). This application uses the image obtained by the coaxial camera and the front camera image for a comprehensive application to ensure the accuracy of meristem identification.

[0063] Step 10: Repeat steps 3 to 9 to complete laser weeding.

[0064] The hardware structure diagram of this application is as follows Figure 1 As shown, it includes a coaxial camera 1 for fine detection, a laser galvanometer 2 and a front camera 3 for rough detection.

[0065] like Figure 2As shown in the figure, the implementation steps of laser weeding technology based on vision guidance are as follows. Take laser weeding in soybean fields as an example:

[0066] Step 1: Collect weed data and pre-train the model

[0067] Step 1.1 Use a data collection vehicle to capture soybean and weed data at 500 mm above the ground under different lighting conditions at random times in the morning, noon, and evening every day.

[0068] Step 1.2: Filter out the images that contain weeds and are clear, use labelme to annotate the weeds in the images, and save the annotated information in the format specified by YOLO and ResNet. The annotated information is divided into training set, validation set, and test set.

[0069] Step 1.3: Perform data enhancement on the image (such as random cropping, rotation, flipping, adjusting brightness and contrast, etc.) to increase data diversity and improve the generalization ability of the model. Perform model training on the processed training data set. Perform transfer learning on the ResNet model, use the parameters of the pre-trained model as the initial parameters, and train on the data of the target task to adapt the model to the target task.

[0070] Step 1.4 exports the trained model weights into a format suitable for deployment, deploys the model in the actual application environment, and optimizes the inference speed.

[0071] Step 2: Deploy the trained model for weed detection

[0072] Step 2.1 calibrates the front camera and the coaxial camera to determine the internal and external parameters of the camera so as to accurately map the information in the 2D image into the 3D world.

[0073] Step 2.2: When the laser weeding robot is running, start the trigger thread of the front camera to send signals according to the set cycle. After each trigger signal is sent, obtain images within the range of 600mm*450mm in the field of view of the front camera, and transmit these images to the weed rough detection thread for processing.

[0074] Step 2.3 resizes the input image to the size required by the model and normalizes and converts the image into a tensor. The preprocessed image is then fed into the YOLO model. The model extracts weed image features through a series of convolutional layers and outputs predictions at multiple scales. The anchor box offsets output by the model are converted into boundary / bounding box coordinates in the actual image. Based on the confidence of the prediction, low-confidence prediction boxes are filtered out. The category probabilities of each prediction box are screened, and only categories with probabilities higher than a certain threshold are retained. The weeds in the front camera are filtered out through the above process.

[0075] Step 3: Use SORT algorithm for multi-target tracking

[0076] Step 3.1 In each frame, the SORT multi-target tracking algorithm uses YOLO to detect the target in the image and obtain the target's bounding box and category confidence.

[0077] Step 3.2 applies a Kalman filter to each tracked target and uses the position and velocity information of the previous frame to predict the position and obtain the predicted position of each target.

[0078] Step 3.3 constructs a cost matrix by calculating the IOU between the detection box and the prediction box, finds the best match between the detection box and the tracking box, completes the target association, assigns a unique ID number in consecutive frames to each weed, and loads the target appearing in the current image frame into the task planning thread in the form of ID number + detection box.

[0079] Each weeding task thread will submit a weeding task request to the task planning thread, and save all target tracking results obtained in the detection program, and quickly index the location and appearance image information of any target through the ID number. The corresponding element in the memory is assigned and updated by the ID number of the weed target detected in each frame. Then the received task requests are processed one by one, and the location, appearance image, detection timestamp and other information of the weed target that has entered the laser working range are sent down, and the information of the target is erased after the sending is completed.

[0080] Multi-target tracking using Sort algorithm

[0081] In each frame, the SORT multi-object tracking algorithm uses YOLO to detect the target in the image and obtain the target's bounding box and category confidence.

[0082] The uniform motion model is used to estimate the position and size of the target in the previous frame in the current frame, and then the target detected by YOLO in the current frame is matched with the target in the previous frame, usually using the IoU metric for matching. The matched targets represent the same object, while the unmatched targets may be newly appeared objects or disappeared objects.

[0083] The optimal state estimate of the matched target is obtained based on the Kalman filter. The Kalman filter combines the observation value of the current frame with the motion model estimation result of the previous frame to obtain the optimal estimate of the target state. This includes information such as position and speed. In this way, the target association is completed and a unique ID number is assigned to each weed in the continuous frame. The target appearing in the current image frame is loaded into the task planning thread in the form of ID number + detection box.

[0084] Step 4: Take the task to deflect the galvanometer and use optical flow field to compensate for the displacement

[0085] Optical flow compensation principle:

[0086] The front camera is triggered quickly and continuously to obtain continuous frame images, and the timestamp of each frame image acquisition is recorded. The high-quality feature points in the image are extracted using the feature point extraction algorithm, and the feature points are located and tracked on two consecutive frames of images. The pixel position vector of the feature point on the two consecutive frames of images is converted into the position vector in the actual physical coordinate system, and then the time difference between the two frames of images is obtained using the corresponding timestamp of the image recorded in 1 to calculate the optical flow speed at the current moment. The sliding window filtering algorithm is used to eliminate the influence of instantaneous speed mutations and improve the stability of the speed obtained by the algorithm. The processing time of the camera's continuous frames to identify the key points of weeds and meristems is calculated. At the current moment, the speed and processing time calculated by the optical flow field are used to compensate for the positioning delay on the coordinates detected by the camera application model.

[0087] It should be noted that the specific implementation is only an explanation and description of the technical solution of the present invention, and cannot be used to limit the scope of protection of the rights. Any partial changes made according to the claims and description of the present invention should still fall within the scope of protection of the present invention.

Claims

1. A laser weeding method based on vision guidance, Features The weeding method is implemented based on a laser weeding robot, which comprises a front camera (3), a coaxial camera (1) and a laser galvanometer (2); The method comprises the following steps: Step 1: calibrate the front camera (3) and the laser galvanometer (2) to obtain a coordinate transformation matrix between the front camera (3) and the laser galvanometer (2); Step 2: Calibrate the coaxial camera (1) and the laser galvanometer (2) to obtain a coordinate transformation matrix between the coaxial camera (1) and the laser galvanometer (2); Step 3: Use the front camera (3) to obtain video data, and input each frame of the video data into a pre-trained first neural network to obtain an output prediction frame and confidence of the weeds, and then select the weed prediction frame with a confidence higher than a threshold as the weed target; Step 4: Based on the weed targets, a multi-target tracking algorithm is used to track the weed targets in the video data, and based on the detection frame of each weed target, the center point coordinates of the detection frame are determined. Then, the y-axis coordinates of the center point coordinates corresponding to the weed targets are sorted from large to small, and the sorting results are used as a processing queue; Step 5: Select the weed targets in sequence according to the order in the processing queue, and obtain the center point coordinates corresponding to the weed targets. Then, use the coordinate transformation matrix between the front camera (3) and the laser galvanometer (2) to obtain the coordinates corresponding to the center point coordinates in the laser galvanometer (2) coordinate system, and deflect the laser galvanometer (2) to the coordinates. Step 6: After the laser galvanometer (2) is deflected, it triggers the coaxial camera (1) to acquire the image within the field of view; Step 7: Crop the weed targets in the detection frame to obtain a cropped image; Step 8: Based on the image acquired by the coaxial camera (1), the image and the cropped image are input into a pre-trained second neural network to obtain the output coordinates of the meristem target point position; Step nine: using the coordinate transformation matrix between the coaxial camera (1) and the laser galvanometer (2), the position of the meristem target point is converted to the corresponding coordinates in the laser galvanometer (2) coordinate system, the laser galvanometer (2) is deflected to the coordinates, and the laser galvanometer (2) is emitted; Step 10: Repeat steps 5 to 9 until all weed targets in the processing queue are selected and laser weeding is completed.

2. A laser weeding method based on vision guidance according to claim 1, Features The pre-trained first neural network is obtained by the following steps: A field image containing weeds is obtained, and crops and weeds in the field image are respectively labeled to construct training data, and the training data is used to train the first neural network.

3. A laser weeding method based on vision guidance according to claim 1, Features The specific steps of step three are: using the front camera (3) to obtain video data, and inputting each frame of the video data into a pre-trained first neural network to obtain output prediction frames and confidences of crops, as well as prediction frames and confidences of weeds, and then selecting the weed prediction frames with confidences higher than a threshold as weed targets.

4. The laser weeding method based on vision guidance according to claim 1, Features The step of tracking the weed targets in the video data using the multi-target tracking algorithm in step 4 includes: Step 41: Select a frame in the video data as the current frame, then use the uniform motion model to estimate the position and size of the weed target in the previous frame in the current frame, and then match the weed target prediction box detected in the current frame with the weed target in the previous frame. The matching is performed using the IoU metric; Step 42: Input the successfully matched target into the Kalman filter to obtain the optimal state estimation of the weed target, wherein the optimal state estimation of the target includes the position and speed of the weed target, and finally the speed of the weed target is used as the speed parameter of the uniform motion model of the next frame.

5. The laser weeding method based on vision guidance according to claim 1, Features The pre-trained second neural network is obtained by the following steps: The key points of the meristem in the weed target are marked, and then the cropped image and the normalized image are used as input, and the marked key points of the meristem are used as output to train the second neural network.

6. A laser weeding method based on vision guidance according to claim 5, Features The second neural network specifically performs the following steps: The second neural network extracts features from the input cropped image and the coaxial camera (1) image respectively, and obtains a weed feature vector with a size of N×1 and a feature vector with a size of M×1 containing weed features and meristem key point features. The weed feature vector and the feature vector containing weed features and meristem key point features are concatenated as final features. The final features are regressed through a multi-layer perceptron to obtain the key points of the weed meristem.

7. The laser weeding method based on vision guidance according to claim 1, Features The method also includes a time t at which the front camera acquires video data 0 The laser galvanometer deflection start time t in step 5 1 The position compensation step of the time delay between is expressed as: Among them, dt 1 t 1 With t 0 The time interval, δx 1 is(t 1 -t 0 ) time in the x direction, δy 1 is(t 1 -t 0 ) time in the y direction, They represent the partial derivatives of the grayscale of the pixel in the image along the x, y, and t directions respectively, and u and v are the velocity vectors of the optical flow along the X-axis and Y-axis respectively.

8. A laser weeding method based on vision guidance according to claim 7, Features The method also includes a time t at which the front camera acquires video data 0 The coaxial camera triggering time t in step 6 2 The position compensation step of the time delay between is expressed as: Among them, dt 2 t 2 With t 0 The time interval, δx 2 is(t 2 -t 0 ) time in the x direction, δy 2 is(t 2 -t 0 ) time in the y direction.

9. The laser weeding method based on vision guidance according to claim 8, Features The method further includes triggering the coaxial camera at the time t 2 The laser emission time t in step 9 3 The position compensation step of the time delay between is expressed as: Among them, dt 3 t 3 With t 2 The time interval, δx 3 is(t 3 -t 2 ) time in the x direction, δy 3 is(t 3 -t 2 ) time in the y direction.

10. The laser weeding method based on vision guidance according to claim 1, Features The first neural network and the second neural network are YOLO models, and the multi-target tracking algorithm is SORT.

Citation Information

Patent Citations

  • Methods and apparatuses for object detection in a scene based on lidar data and radar data of the scene

    CN111201451A

  • Weeding robot, planning method and device for weeding path of weeding robot and medium

    CN113597874A

  • Weed positioning method based on binocular vision

    CN116721149A

  • Weed center segmentation recognition and laser weeding device

    CN118230160A

Cited By

  • Intelligent laser weeding dynamic fine positioning method and system based on machine vision

    CN119515963A

  • Wide laser weeding system based on galvanometer-linear motor cooperative motion compensation and control method

    CN120858958A

  • Solar laser weeding machine and weeding method

    CN121844827A

  • Machine vision-based dynamic fine positioning method and system for smart laser weeding

    WO2026114051A1