Laser weeding system and method based on coordinate regression and visual control

Abstract

A laser weeding system and method based on coordinate regression and visual control, comprising: an image acquisition module, an image processing module, a weed growth point coordinate regression prediction module, a coordinate transformation module, a visual servo control module, a galvanometer drive module, a red guide laser module and a main laser module, the present application can improve the positioning accuracy of weed growth point and the laser hit rate, reduce the risk of misinjury to crops, has the advantages of non-contact, automation, strong real-time and good adaptability, and is suitable for precise laser weeding operation on greenhouse, open field and intelligent agricultural machinery platform.

Description

Technical Field

[0001] This invention relates to a technology in the field of agricultural automation, specifically a laser weeding method based on coordinate regression and visual servo control for locating weed growth points and targeting them. Background Technology

[0002] Existing laser weeding systems still have the following problems in practical applications: weed target recognition is mostly limited to the whole plant or leaf level, lacking high-precision positioning of the "growth point," a key inactivation site; factors such as changes in field light, weed obstruction, and complex postures lead to unstable positioning; existing weed positioning models based on recognition or segmentation can only predict the centroid position of the weed, usually corresponding to non-lethal areas such as leaves and stems, and cannot guide the laser to burn the core growth area of ​​the weed, often resulting in energy waste and low work efficiency. At the same time, the target accuracy is still sensitive to ground undulations, and millimeter-level deviations can affect the weeding effect; expanding the spot-cutting range will inevitably prolong the operation time, thus failing to meet the needs of dynamic operations. Summary of the Invention

[0003] This invention addresses the problems of inaccurate positioning, unstable target aiming, and accidental crop damage in existing laser weeding systems. It proposes a laser weeding system and method based on coordinate regression and visual control, which can improve the positioning accuracy of weed growth points and the laser hit rate, reduce the risk of accidental crop damage, and has the advantages of being non-contact, automated, real-time, and adaptable. It is suitable for precision laser weeding operations in greenhouses, open fields, and on intelligent agricultural machinery platforms.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a laser weeding system based on coordinate regression and visual control, comprising: an image acquisition module, an image processing module, a weed growth point coordinate regression prediction module, a coordinate transformation module, a visual servo control module, a galvanometer driving module, a red guide laser module, and a main laser module. The image acquisition module acquires RGB images of the ground working area in real time; the image processing module performs noise reduction processing on the RGB images containing the weed targets and the red guide laser spot, and locates the coordinates of the guide laser spot for subsequent visual servo control; the weed growth point coordinate regression prediction module also uses the RGB image as input to perform weed removal... The normalized coordinate regression calculation of the grass growth point; the coordinate transformation module converts the normalized coordinates into image pixel coordinates according to the image size, and calculates the error according to the pixel coordinates of the guide spot. Then, based on the galvanometer calibration results, the pixel coordinates and error are converted into galvanometer coordinates and increments; the visual servo control module calculates the new galvanometer coordinates according to preset control parameters such as the scaling factor, and submits them to the galvanometer drive module to control the galvanometer to deflect. If the error between the old and new coordinates is less than the preset threshold, the visual servo control module turns off the red guide laser module and turns on the main laser module according to preset power, time and other parameters to perform a weeding operation.

[0006] The visual servoing control adopts IBVS based on image feature error, or PBVS based on position error, or a hybrid visual servoing strategy that combines the two.

[0007] This invention relates to a laser weeding method based on the above-mentioned system. The method acquires field weed images through an image acquisition module, uses a target detection and coordinate regression prediction network to coarsely locate the weeds and their core growth points, and performs fine coordinate regression to obtain the predicted coordinates of the laser burning point in the image coordinate system. These predicted coordinates are then mapped to the working coordinate system of the actuator. A visual servo control strategy is employed, using a visible light band guiding laser beam to perform real-time closed-loop correction of the laser emission mechanism, ensuring the laser beam is continuously aligned with the weed growth point. Laser emission is triggered after the target error threshold is met, achieving precise burning and weeding of the weed growth point.

[0008] The method specifically includes:

[0009] Step 1: The image acquisition module acquires field images containing crops and weeds, and transmits the images to the image processing module for preprocessing; the preprocessed images are then input into the weed growth point coordinate regression prediction module, which outputs the location of the weed growth points and the corresponding confidence level.

[0010] Step two: In the training phase of the prediction model based on deep neural networks, manually labeled weed growth points are used as supervision information, and the model is trained using the dataset.

[0011] The dataset includes: an image in WxH format and a list of pixel coordinates (x, y) of the corresponding weed growth points in the image.

[0012] The dataset has undergone data augmentation processing, including random brightness perturbation, rotation transformation, scaling, blur enhancement, and color perturbation.

[0013] Step 3: Convert the image coordinates of the weed growth points into control quantities that the galvanometer system can execute. Based on the camera calibration parameters, the spatial installation relationship between the camera and the galvanometer, and the calibration results of the working plane, the image plane coordinates are mapped to the working plane coordinates, and then further converted into galvanometer driving voltage or deflection angle.

[0014] Preferably, low-pass filtering, dead-zone compensation, and time delay compensation are added during the conversion process to improve the robustness of the galvanometer to high-frequency jitter and image noise.

[0015] Step four: In the online phase, the PID parameters are empirically tuned or adjusted online based on an adaptive algorithm.

[0016] The prediction model based on deep neural networks is trained using an image dataset with growth point annotations, and the loss function includes coordinate regression loss.

[0017] In addition to the laser generator, the laser emitting mechanism also includes at least one of a two-dimensional galvanometer, a gimbal, a micro robotic arm, or a mobile platform, for the purpose of correcting the direction of the laser beam.

[0018] The method described above introduces motion compensation, time synchronization, and predictive compensation mechanisms during the target engagement process to compensate for errors caused by platform vibration, target micro-motion, and system time delay.

[0019] The laser parameters include power and irradiation time, and are adjusted according to the weed species, growth stage, and environmental conditions.

[0020] Technical effect

[0021] This invention employs a neural network structure designed based on coordinate regression to achieve end-to-end positioning of the core growth area of ​​weeds, and introduces visible light-assisted visual servo closed-loop control to achieve automatic target alignment for laser weeding. Attached Figure Description

[0022] Figure 1 This is a flowchart of the present invention;

[0023] Figure 2 This is a schematic diagram of the system topology module structure for an example.

[0024] Figure 3 This is a schematic diagram illustrating the effect of regression prediction of weed growth point coordinates in this invention. Detailed Implementation

[0025] like Figure 2 As shown in the figure, this embodiment relates to a laser weeding system based on coordinate regression and vision control, including: an image acquisition module, an image processing module, a weed growth point coordinate regression prediction module, a coordinate transformation module, a visual servo control module, a galvanometer driving module, a red guiding laser module, and a main laser module. The image acquisition module outputs a 30fps, 1080p video stream from an RGB camera; the image processing module performs distortion reduction and noise reduction on the video stream, and locates the red guiding light spot in the image with a soil and vegetable field background in the video stream; the weed growth point coordinate regression prediction module predicts two-dimensional normalized coordinates based on a deep neural network architecture; the coordinate transformation module converts the normalized coordinates output by the prediction module into image pixel coordinates according to the image size in the video stream, and converts the image pixel coordinates into galvanometer working coordinates according to the galvanometer calibration results; the visual servo control module continuously calculates the galvanometer input coordinates using a PID controller or other classic closed-loop controller based on the positioning error.

[0026] The galvanometer driving module employs, but is not limited to, a two-dimensional galvanometer system with an adjustable optical path for target aiming.

[0027] The red guiding laser module described herein employs, but is not limited to, a low-power visible light laser generator.

[0028] The main laser module employs, but is not limited to, a high-power laser generator.

[0029] like Figure 1 As shown, this embodiment illustrates a laser weeding method based on the aforementioned system, comprising:

[0030] Step 1: The image acquisition module acquires field images containing crops and weeds, and transmits the images to the image processing module for preprocessing. Preprocessing includes distortion correction, white balance adjustment, noise suppression, brightness normalization, and cropping. The preprocessed images are then input into the weed growth point coordinate regression prediction module, which outputs the location of the weed growth points and their corresponding confidence scores.

[0031] The growth point refers to the meristematic tissue zone, stem area, or other key parts of the weed that are more sensitive to laser burning. Compared with the traditional method of targeting the entire plant center, this invention directly locates the growth point, which can significantly improve the targeting and inactivation effect of weed control.

[0032] The coordinate regression prediction module adopts a relative coordinate regression network. Its input is a preprocessed weed image, and its output is the two-dimensional normalized offset (Δx, Δy) of the growth point relative to the upper left corner of the image, satisfying Δx∈[0,1], Δy∈[0,1].

[0033] Furthermore, the coordinate regression prediction module may include a feature extraction backbone network, a multi-scale feature fusion module, a coordinate regression head, and a confidence evaluation branch. The backbone network is used to extract texture, contour, and local geometric features from weed image patches; the multi-scale feature fusion module is used to enhance the model's adaptability to weeds of different sizes and growth stages; the coordinate regression head is used to output the position coordinates of the growth points; and the confidence evaluation branch is used to evaluate the reliability of the prediction results.

[0034] Step 2: In the training phase of the deep neural network-based prediction model, manually labeled weed growth points are used as supervision information. The model is trained using a dataset containing different lighting conditions, shooting distances, weed species, seasons or growth cycles, and occlusion states. To improve the model's generalization performance, data augmentation processing such as random brightness perturbation, rotation transformation, scaling, blur enhancement, and color perturbation can be applied to the training data.

[0035] The training described uses the following loss function. L coord For coordinate regression loss, Smooth L1 loss is preferred; L reg For regularization terms; The loss weight coefficients; coordinate regression loss N is the number of samples. Predict coordinates for the model. The coordinates are accurately labeled.

[0036] Step 3: Convert the image coordinates of the weed growth points into control quantities that the galvanometer system can execute: Based on the camera calibration parameters, the spatial installation relationship between the camera and the galvanometer, and the calibration results of the working plane, map the image plane coordinates to the working plane coordinates, and then further convert them into galvanometer driving voltage or deflection angle, specifically including:

[0037] 3.1 Let the coordinates of the weed growth point on the working plane be [x...]. g , y g The galvanometer control input is [u] x , u y ],but Where K is the scaling transformation matrix and b is the bias vector. The above mapping relationship can be obtained through offline calibration or updated online during system operation to improve the mapping accuracy under different working distances and installation error conditions.

[0038] 3.2 To achieve precise target aiming with the main laser, a galvanometer system drives a low-power red guide laser beam to form a visible light point. The position of this point is adjusted in real-time using a proportional-integral-derivative (PID) visual servoing method based on image error, gradually aligning the red guide laser point with the weed growth point. The red guide laser beam is used only for aiming and does not participate in the weeding process. Specifically, it includes:

[0039] 3.2.1 The camera acquires images of the current field of view in real time;

[0040] 3.2.2 Extract the position (x, y) of the red guide laser point in the image. l , y l );

[0041] 3.2.3 Calculate the image error between the red guiding laser point and the weed growth point. ;

[0042] 3.2.4 Input the error into the dual-channel PID controller, which outputs the control quantities of the galvanometer in the X and Y axes respectively. ,in: and These are the control inputs for the galvanometer in two directions, respectively. , , These are the proportional, integral, and derivative control parameters, respectively.

[0043] 3.2.5 The galvanometer adjusts the laser spot position according to the control parameters;

[0044] 3.2.6 Repeat steps 3.2.1-3.2.5 until the error meets the threshold condition.

[0045] Step 4: In the online phase, to improve system response speed and reduce oscillations, the PID parameters are empirically tuned or adjusted online based on an adaptive algorithm. Specifically, when the following conditions are met... , When the system determines that the red guide laser beam is aligned with the weed growth point, the control module outputs a trigger signal to activate the main laser to burn the target location. and This is a preset target threshold.

[0046] The main laser irradiation time, pulse power, and repetition frequency can be set or adaptively adjusted according to the weed species, growth stage, target size, and environmental conditions.

[0047] Through practical application experiments, the weed growth point localization algorithm based on coordinate regression of this invention was tested on typical weed samples such as crabgrass, purslane, and ecliptic snake under laboratory conditions. The hardware and software environment was equipped with an NVIDIA GeForce RTX 3070 graphics card, Windows 10 operating system, Intel Core i7 processor, 32 GB memory, 1080p resolution RGB camera, dual-axis galvanometer controller and 1064nm laser. The data were collected by a field mobile platform moving at a constant speed. The algorithm was tested on laboratory conditions.

[0048] Table 1: Results of the experiment on locating weed growth points using a mobile platform moving at a constant speed in the field.

[0049] Table 2: Experimental Results of Visual Servo Target Alignment with a Single Laser

[0050] Compared with existing technologies, this invention achieves precise positioning of weed growth points through a coordinate regression prediction model. Compared with the existing technology that calculates the burn point from the centroid of the weed mask area, this significantly reduces the average positioning error after target detection and the risk of misalignment caused by only detecting the entire weed area. Through visual servo closed-loop control, the direction is corrected in real time using the guiding laser spot, which can compensate for the misalignment between the actual galvanometer working plane and the calibration plane caused by platform vibration, uneven ground, and slight movement of the weed target, significantly reducing the target error and improving the success rate of laser weeding. At the same time, the high-precision point-to-point target alignment avoids the area scanning working mode, which can effectively reduce the thermal damage of the laser to nearby crops and reduce the seedling damage rate. Its rapid response further improves the stability and robustness of the system in complex dynamic field scenarios.

[0051] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A laser weeding system based on coordinate regression and visual control, characterized in that, include: The system comprises an image acquisition module, an image processing module, a weed growth point coordinate regression prediction module, a coordinate transformation module, a visual servo control module, a galvanometer drive module, a red guiding laser module, and a main laser module. Specifically: the image acquisition module acquires real-time RGB images of the ground work area; the image processing module performs noise reduction on the RGB images containing weed targets and the red guiding light spot, and locates the coordinates of the guiding light spot for subsequent visual servo control; the weed growth point coordinate regression prediction module also uses the RGB image as input to perform normalized coordinate regression calculations for the weed growth points; the coordinate transformation module converts the normalized coordinates into image pixel coordinates based on the image size, calculates the error based on the pixel coordinates of the guiding light spot, and then converts the pixel coordinates and error into galvanometer coordinates and increments based on the galvanometer calibration results; the visual servo control module calculates new galvanometer coordinates based on preset control parameters such as a scaling factor, submits the result to the galvanometer drive module to control the galvanometer deflection, and if the error between the old and new coordinates is less than a preset threshold, the visual servo control module shuts down the red guiding laser module and turns on the main laser module according to preset power, time, and other parameters to perform a weeding operation; The visual servoing control adopts an IBVS based on image feature error, a PBVS based on position error, or a hybrid visual servoing strategy that combines the two.

2. A laser weeding method based on coordinate regression and visual control according to the system of claim 1, characterized in that, Images of field weeds are acquired through an image acquisition module. A target detection and coordinate regression prediction network is used to coarsely locate the weeds and their core growth points and perform fine coordinate regression. After obtaining the predicted coordinates of the laser burning point in the image coordinate system, the predicted coordinates are mapped to the working coordinate system of the actuator. A visual servo control strategy is adopted, and a guide laser beam in the visible light band is used to perform real-time closed-loop correction of the laser emission mechanism, so that the laser beam is continuously aimed at the weed growth point. The laser emission is triggered after the target error threshold is met, so as to achieve precise burning and weeding of the weed growth point.

3. The laser weeding method based on coordinate regression and visual control according to claim 2, characterized in that, include: Step 1: The image acquisition module acquires field images containing crops and weeds, and transmits the images to the image processing module for preprocessing; The preprocessed image is input into the weed growth point coordinate regression prediction module, which outputs the location of the weed growth point and the corresponding confidence level. Step 2: In the training phase of the prediction model based on deep neural networks, manually labeled weed growth points are used as supervision information, and the model is trained using the dataset. Step 3: Convert the image coordinates of the weed growth point into control quantities that can be executed by the galvanometer system: Based on the camera calibration parameters, the spatial installation relationship between the camera and the galvanometer, and the calibration results of the working plane, map the image plane coordinates to the working plane coordinates, and then further convert them into galvanometer driving voltage or deflection angle. Step four: In the online phase, the PID parameters are empirically tuned or adjusted online based on an adaptive algorithm.

4. The laser weeding method based on coordinate regression and visual control according to claim 3, characterized in that, The dataset includes: an image in WxH format and a list of pixel coordinates (x, y) of the corresponding weed growth points in the image. The dataset has been processed by random brightness perturbation, rotation transformation, scale scaling, blur enhancement and color perturbation.

5. The laser weeding method based on coordinate regression and visual control according to claim 3, characterized in that, Low-pass filtering, dead-zone compensation, and time delay compensation are added during the conversion process to improve the robustness of the galvanometer to high-frequency jitter and image noise.

6. The laser weeding method based on coordinate regression and visual control according to claim 3, characterized in that, The aforementioned prediction model based on deep neural networks is trained using an image dataset with growth point annotations, and the loss function includes coordinate regression loss, specifically: L coord For coordinate regression loss, Smooth L1 loss is preferred; L reg For regularization terms; The loss weight coefficients; coordinate regression loss N is the number of samples. Predict coordinates for the model. The coordinates are accurately labeled.

7. The laser weeding method based on coordinate regression and visual control according to claim 3, characterized in that, Step three specifically includes: 3.1 Let the coordinates of the weed growth point on the working plane be [x...]. g , y g The galvanometer control input is [u] x , u y ],but Where K is the scaling transformation matrix and b is the bias vector. The above mapping relationship can be obtained through offline calibration or updated online during system operation to improve the mapping accuracy under different working distances and installation error conditions; 3.2 To achieve precise target aiming with the main laser, a galvanometer system is used to drive a low-power red guide laser beam to form a visible light spot. The position of this spot is then adjusted in real-time using a visual servoing method based on image error proportional-integral-derivative (PID), gradually aligning the red guide laser spot with the weed growth point. The red guide laser beam is used only for aiming and does not participate in the weeding process.

8. The laser weeding method based on coordinate regression and visual control according to claim 7, characterized in that, Step 3.2 specifically includes: 3.2.1 The camera acquires images of the current field of view in real time; 3.2.2 Extract the position (x, y) of the red guide laser point in the image. l , y l ); 3.2.3 Calculate the image error between the red guiding laser point and the weed growth point. ; 3.2.4 Input the error into the dual-channel PID controller, which outputs the control quantities of the galvanometer in the X and Y axes respectively. ,in: and These are the control inputs for the galvanometer in two directions, respectively. , , These are the proportional, integral, and derivative control parameters, respectively. 3.2.5 The galvanometer adjusts the laser spot position according to the control parameters; 3.2.6 Repeat steps 3.2.1-3.2.5 until the error meets the threshold condition.

9. The laser weeding method based on coordinate regression and visual control according to claim 3, characterized in that, When satisfied , When the system determines that the red guide laser beam is aligned with the weed growth point, the control module outputs a trigger signal to activate the main laser to burn the target location. and This is a preset target threshold.

Images