Unmanned aerial vehicle cotton topping method and system based on machine vision

Abstract

The invention discloses an unmanned aerial vehicle cotton topping method and system based on machine vision. According to the method, an unmanned aerial vehicle is controlled to fly in an interlaced operation mode, an airborne vision module is used for collecting cotton terminal bud images of adjacent non-operation rows, and image stability is judged; after terminal buds are recognized, the spatial positions of the terminal buds are determined, time delay is calculated, and a spraying instruction is generated; and controlling the pesticide application module to accurately apply pesticide to the terminal buds according to the instruction. According to the interlaced operation mode, the downwash wind field of the unmanned aerial vehicle acts on the current flying row, and disturbance to cotton plants in adjacent non-operation rows is reduced to the maximum extent. The system comprises an unmanned aerial vehicle platform, a visual module, a pesticide application module and a flight control and processing module. Spraying time is synchronized through a delay compensation method, and a binocular vision positioning technology is combined, so that positioning errors caused by interference and movement of a downwash wind field of the unmanned aerial vehicle are effectively overcome, the recognition accuracy and the pesticide application precision are remarkably improved, and efficient cotton topping is realized.

Description

Technical Field

[0001] This invention relates to the field of precision agriculture technology and intelligent agricultural equipment, and in particular to a method and system for topping cotton using a drone based on machine vision. Background Technology

[0002] Topping is a crucial step in cotton production. By removing apical dominance, it encourages nutrients to be concentrated on the bolls, effectively improving cotton yield and quality. Traditional topping methods mainly include manual and mechanical topping. Manual topping is labor-intensive, inefficient, and the quality varies depending on the individual. While mechanical topping improves efficiency, it causes greater damage to the cotton plant and has significant physical limitations.

[0003] In recent years, with the development of precision agriculture and intelligent equipment technology, drones have been widely used in farmland operations due to their high efficiency and flexibility. Current research on drone-based agricultural spraying mainly focuses on canopy-scale growth monitoring or large-area uniform spraying, with less research on precision operations. However, because drones are constantly moving at high speeds, there is an unavoidable system delay from image acquisition, processing, and recognition to final spraying, resulting in delayed spraying positions and inaccurate targeting of the terminal buds, ultimately affecting topping effectiveness and wasting pesticides. Furthermore, the strong downwash airflow generated by drones during operation causes significant shaking of cotton plants, leading to blurred images and greatly reducing the accuracy and reliability of visual recognition algorithms.

[0004] Therefore, there is an urgent need for a drone-based cotton topping technology that can effectively overcome the interference of the downwash wind field and achieve dynamic and precise target application of pesticides, in order to solve the problems of low identification accuracy and poor application precision in the existing operation methods. Summary of the Invention

[0005] This invention provides a machine vision-based method and system for topping cotton plants using drones, aiming to solve problems such as low recognition accuracy caused by cotton plant swaying due to the downwash airflow of drones and poor spraying accuracy caused by system delays in the existing technology.

[0006] The present invention provides a machine vision-based method for topping cotton plants using a drone, characterized by the following steps:

[0007] Step S1: Based on the cotton field planting pattern, set the drone flight path. The drone flies along the cotton field at a height of 1.5-2m above the ground, working in alternating rows. It collects images of the cotton buds below through the vision module mounted on the drone.

[0008] Step S2: Process the acquired apical bud image using the recognition model pre-deployed on the UAV's onboard computer, analyze the image clarity and feature point jitter, and continue execution when the image is determined to be stable;

[0009] Step S3: Determine the spatial position of the apical bud in the image, calculate the delay, and generate the corresponding spraying command;

[0010] Step S4: According to the spraying command, control the application module to apply the pesticide to the apical bud.

[0011] The drone operates in an inter-row manner, so that the downwash air field it generates mainly affects the currently flying rows that have been operated or the areas without cotton planting, thereby minimizing the disturbance to cotton plants in adjacent unoperated rows and ensuring the accuracy of image recognition.

[0012] Step S1 includes: planning the drone flight path based on the cotton field planting row numbers, adopting an inter-row operation mode, controlling the drone to fly in row number order, and limiting it to identifying and applying pesticides to adjacent unoperated rows; for example, when the drone flies over row N, it identifies and applies pesticides to cotton plants in row N+1; if operation is needed on row 1, the drone is controlled to fly from the area outside row 1 where there are no cotton plants. The drone collects images of cotton apical buds during flight.

[0013] Step S2 requires pre-training a bud recognition model, which is then deployed on the drone's onboard computer to process the collected cotton bud images during topping operations and to assess the stability of the images. If the image is determined to be stable, subsequent operations proceed. The specific steps for training the cotton bud recognition model include:

[0014] Step S2a: Construct a dedicated dataset for cotton apical buds. Specifically, collect images of cotton apical buds under different lighting conditions and different interference scenarios, process the images, expand the dataset size through data augmentation techniques, and divide it into training set, test set and validation set according to the proportions.

[0015] Step S2b: Based on the convolutional neural network structure, transfer learning is performed using pre-trained weights to obtain the final model weights;

[0016] Step S2c: Convert the trained model weights into an open neural network exchange format, and generate a high-performance inference engine suitable for embedded platforms through an optimized toolchain, which is then deployed on the drone's onboard computer.

[0017] Step S2d: Perform performance verification on the optimized model to ensure that its recognition accuracy and inference speed meet the requirements of real-time field operations.

[0018] Step S3 includes:

[0019] Step S3a: Based on the pixel coordinates of the cotton apical bud obtained in step S2, the depth information of the apical bud is calculated using the stereo matching principle of the binocular vision system.

[0020] Step S3b: Using the intrinsic parameter matrix obtained from camera intrinsic parameter calibration, transform the identified apical pixel coordinates (u, v) to three-dimensional coordinates P in the camera coordinate system. camera =(X c Y c Z c ) T It is calculated using the following formula:

[0021]

[0022] Among them, Z c C represents the apical bud depth value calculated in step S3a. x and C y Let f be the coordinates of the camera's optical center in the image pixel coordinate system. x and f y These are the focal length values ​​of the camera in the x and y directions.

[0023] Step S3c: Using the rotation matrix and translation vector obtained from the camera extrinsic calibration, adjust the coordinates P in the camera coordinate system. camera Coordinates P transformed into the UAV body coordinate system body =(X b Y b Z b ) T The conversion relationship is defined by the following formula:

[0024] P body =R -1 ·(P camera -T)

[0025] Where R is the rotation matrix and T is the translation vector.

[0026] Furthermore, based on the coordinates of the cotton bud relative to the UAV body coordinate system obtained from the above steps, the current flight speed of the UAV, the image processing delay of the vision module, and the mechanical response delay of the pesticide application module, the pesticide application delay compensation time is calculated, and finally, the corresponding spraying command is generated so that the execution time of the spraying command matches the time when the UAV flies to the top of the bud.

[0027] The application delay compensation time T is calculated using the following formula:

[0028] T = D / V + T processing +T mechanical +T actuation

[0029] Where D is the pre-trigger three-dimensional straight-line distance between the UAV and the spray target point, V is the actual flight speed of the UAV, and T is the distance between the UAV and the target point. processing For the system's image perception and decision-making delay, Tmechanical For mechanical response delay, T actuation This refers to the flight time of the drug solution.

[0030] Step S4, based on the generated spraying command, controls the application module to perform precise targeted application of pesticide to the identified and located cotton apical buds. The application module receives a trigger signal from the flight controller and sprays pesticide at the time and location specified in the command through the nozzle, thereby achieving pesticide coverage of the apical buds. At the same time, by adjusting the nozzle angle and pesticide flow rate, it ensures that the pesticide is concentrated on the apical bud, minimizing drift and waste, and improving pesticide utilization and topping operation effectiveness.

[0031] Accordingly, this invention also provides a machine vision-based drone cotton topping system, comprising: a drone flight platform: employing a multi-rotor flight platform equipped with a lidar sensor and an inertial measurement unit for acquiring drone flight attitude information; a vision module: equipped with a binocular depth camera for acquiring the position information of the cotton apical bud; a pesticide application module: including a pesticide storage unit, a power delivery unit, and an atomizing spraying unit for performing targeted pesticide application to the apical bud; and a flight control and processing module: including an onboard computer and a flight controller; the onboard computer is responsible for image data processing and apical bud recognition calculation, while the flight controller is responsible for controlling the flight attitude and precisely triggering the pesticide application module; the two interact via a serial communication bus to achieve coordinated control of asynchronous environmental perception and synchronous pesticide application triggering, ensuring the accuracy of the topping operation. The vision module and pesticide application module are generally balanced relative to the drone flight platform and flight control and processing module, with the nozzle positioned 20-100cm horizontally from the drone and 0-100cm below the drone.

[0032] The present invention has the following beneficial effects:

[0033] By adopting an innovative inter-row operation mode, interference from the drone's downwash air field on machine vision recognition is effectively avoided, significantly improving the accuracy of bud identification and pesticide application precision, and achieving efficient and precise chemical topping of cotton. The method of judging image stability before operation is adopted to improve the accuracy of operation. The multi-dimensional delay compensation method solves the problem of precise target aiming under mobile platform, effectively improving the accuracy of pesticide application. The system architecture of asynchronous perception and synchronous control is designed to balance the requirements of processing power and real-time operation. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0035] Figure 1 This is a flowchart of a machine vision-based drone cotton topping method according to an embodiment of the present invention;

[0036] Figure 2 This is a structural diagram of a machine vision-based drone cotton topping system according to an embodiment of the present invention;

[0037] Figure 3 This is a flowchart of a machine vision-based cotton topping drone operation in an embodiment of the present invention;

[0038] Figure 4 This is a flowchart illustrating the training process of a cotton apical bud recognition model based on deep learning, as described in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Figure 1 This is a flowchart of a machine vision-based drone-based cotton topping method according to an embodiment of the present invention, as shown below. Figure 1 The method includes the following steps:

[0041] Step S1: Plan the drone flight path based on the cotton field planting row numbers, adopting an inter-row operation mode. Control the drone to fly sequentially according to the row numbers, with a flight altitude of 1.5-2m above the ground, and limit its operation to identifying and applying pesticides to adjacent unoperated rows; for example, when the drone flies over row N, it identifies and applies pesticides to cotton plants in row N+1; if operation is needed on row 1, control the drone to fly from the area outside row 1 where there are no cotton plants. During flight, the drone uses its onboard vision module to collect images of the cotton buds below.

[0042] Step S2: The apical bud recognition model needs to be trained in advance and deployed on the drone's onboard computer to process the collected cotton apical bud images during topping operations. The model also needs to assess the stability of the collected images; if the image is determined to be stable, subsequent operations can proceed. Figure 4 The training steps for the cotton apical bud recognition model include:

[0043] Step S2a: Construct a dedicated dataset for cotton apical buds. Specifically, collect images of cotton apical buds under different lighting conditions (such as morning, noon, and evening) and different interference scenarios (such as leaf shading). Use Labelimg software to accurately label the cotton apical buds in the collected images. Expand the dataset size through data augmentation techniques (mirroring, rotation, scaling, and translation). Divide the dataset into training set, validation set, and test set in a ratio of 7:2:1.

[0044] Step S2b: Use the YOLOv5s recognition algorithm, set the initial learning rate to 0.01, the training batch size to 16, train for a total of 300 epochs, and use the YOLOv5s algorithm pre-trained weights for transfer learning to obtain the final model weights.

[0045] Step S2c: Convert the final weighted PyTorch model to ONNX format, and use Intel OpenVINO Model Optimizer to convert the ONNX model to IR format. Use INT8 quantization for lightweighting, and control the accuracy loss within an acceptable range by calibrating the dataset (apical bud recognition mAP decrease ≤2%). The final model size is controlled within 5MB and deployed on the drone's onboard computer.

[0046] Step S2d: The INT8 quantized model was tested in practice: the input image size was 640×640, and the test dataset covered real field scenes such as sunny and cloudy days (a total of 500 labeled images). The final apical bud recognition accuracy reached mAP≥90%, and the inference speed was ≥20 frames / second, which can ensure the accuracy and timeliness of the drone's real-time bud topping operation in the field.

[0047] Furthermore, after the onboard computer receives the image data in real time, it first performs image stability analysis: the ORB feature point detection algorithm is used to calculate the pixel displacement of feature points between consecutive frames. When the displacement exceeds a threshold of 5 pixels, the image is determined to be unstable and the operation is suspended. When the image is stable, the cotton bud is identified and the pixel coordinates of the bud in the image are obtained.

[0048] Step S3: Determine the spatial location of the cotton bud in the image, calculate the delay, and generate the corresponding spraying command. The steps for calculating the three-dimensional spatial coordinates of the cotton bud include:

[0049] Step S3a: Based on the cotton apical bud pixel coordinates obtained in step S2, the disparity d is obtained using the stereo matching principle of a binocular vision system. The depth information of the apical bud is then calculated using the following formula:

[0050]

[0051] Where B is the baseline distance of the binocular cameras, and Z... c This represents the depth of the apical bud.

[0052] Step S3b: After obtaining depth Z c Then, using the inverse transformation of the camera intrinsic parameter matrix K, the identified bud pixel coordinates (u, v) are transformed into three-dimensional coordinates P in the camera coordinate system. camera =(X c Y c Z c ) T It is calculated using the following formula:

[0053]

[0054] Expand

[0055]

[0056] Among them, C x and C y Let f be the coordinates of the camera's optical center in the image pixel coordinate system. x and f y These are the focal length values ​​of the camera in the x and y directions.

[0057] Step S3c: Using the rotation matrix and translation vector obtained from the camera extrinsic calibration, adjust the coordinates P in the camera coordinate system. camera Coordinates P transformed into the UAV body coordinate system body =(X b Y b Z b ) T The conversion relationship is defined by the following formula:

[0058] P body =R -1 ·(P camera -T)

[0059] Where R is the rotation matrix and T is the translation vector.

[0060] In addition, the camera intrinsic parameter matrix K, the baseline distance B of the stereo camera, the rotation matrix R, and the translation vector T need to be obtained in advance through camera intrinsic and extrinsic parameter calibration.

[0061] Furthermore, based on the coordinates of the cotton bud relative to the UAV body coordinate system obtained from the above steps, the current flight speed of the UAV, the image processing delay of the vision module, and the mechanical response delay of the pesticide application module, the pesticide application delay compensation time is calculated, and finally, the corresponding spraying command is generated so that the execution time of the spraying command matches the time when the UAV flies to the top of the bud.

[0062] The application delay compensation time T is calculated using the following formula:

[0063] T = D / V + T processing +T mechanical +T actuation

[0064] Where D is the pre-trigger three-dimensional straight-line distance between the UAV and the spray target point, V is the actual flight speed of the UAV, and T is the distance between the UAV and the target point. processing For the system's image perception and decision-making delay, T mechanical For mechanical response delay, T actuation This refers to the flight time of the drug solution.

[0065] The formula for calculating the pre-trigger three-dimensional straight-line distance D between the UAV and the spraying target point is as follows:

[0066]

[0067] Among them, X b Y b Z b The coordinates of the cotton bud are given in the UAV body coordinate system.

[0068] The actual flight speed V of the drone is taken as the velocity component in the direction of the drone's forward movement, and is obtained in real time by the state estimator of the flight controller.

[0069] Image perception and decision delay T of the system processing This includes image transmission delay, neural network inference delay, coordinate transformation and decision calculation delay. This value can be calibrated to a constant after system initialization, with a total value of approximately 40ms.

[0070] Mechanical response delay T mechanical This includes circuit response delay and mechanical start-up delay, which is approximately 15ms.

[0071] Drug flight time T actuation This time is determined by the nozzle height H above the ground and the average droplet velocity V. droplet The decision is made, and the calculation formula is as follows:

[0072]

[0073] Step S4: Based on the spraying command issued by the flight controller, the application module is controlled to perform precise spraying on the identified and located cotton buds. The specific process is as follows: The spraying command is a digital control signal issued by the flight controller, which includes the body coordinates of the target bud, the preset spray volume, and the precise delay compensation time for the nozzle movement. After receiving the command, the application module drives the actuator to operate. Specifically, the application module includes a nozzle, a water pump, a solenoid valve, and a medicine tank. The nozzle achieves pesticide coverage on the buds through pulse spraying, with a single spray duration of 50-200ms.

[0074] Figure 3 This is a flowchart illustrating a machine vision-based cotton topping drone operation in an embodiment of the present invention, which further refines the process. Figure 1 This cyclical mechanism greatly enhances the system's robustness in complex field environments.

[0075] The present invention also relates to a system for implementing the above method, having the following structure: Figure 2As shown. The system includes a quadcopter drone platform (1); a vision module (2): a binocular depth camera RealSense D435i; a spraying module: including a nozzle (3), a water pump (4), a solenoid valve and a medicine tank; and a flight control and processing module: a flight controller (5) and an onboard computer (6). The flight control and processing module consists of an asynchronous perception channel composed of a high-performance onboard computer (Intel NUC12WSKI70Z), responsible for computationally intensive image recognition and decision-making tasks; and a synchronous control channel composed of a flight controller (Pixhawk 6C Mini), responsible for real-time flight stability and precise spraying command triggering. The two work together to complete the closed loop of perception, decision-making and control. Among them, the vision module (2) and the spraying module are generally balanced relative to the drone flight platform (1) and the flight control and processing module. The nozzle is 20-100cm away from the drone in the horizontal direction and 0-100cm away from the drone below.

[0076] In summary, this invention effectively solves the problems of downwash wind field interference and precise pesticide application on mobile platforms by using an interlaced operation strategy, a delay compensation method, an image stability determination method, and an asynchronous sensing and synchronous control system architecture, thus achieving high precision, high efficiency, and intelligent cotton topping by drones.

Claims

1. A machine vision-based method for topping cotton plants using a drone, characterized in that, The method includes the following steps: Step S1: Based on the cotton field planting pattern, set the drone flight path. The drone flies along the cotton field at a height of 1.5-2m above the ground, working in alternating rows. It collects images of the cotton buds below through the vision module mounted on the drone. Step S2: Process the acquired apical bud image using the recognition model pre-deployed on the UAV's onboard computer, analyze the image clarity and feature point jitter, and continue execution when the image is determined to be stable; Step S3: Determine the spatial position of the apical bud in the image, calculate the delay, and generate the corresponding spraying command; Step S4: According to the spraying command, control the application module to apply the pesticide to the apical bud. The drone operates in an inter-row manner, so that the downwash air field it generates mainly affects the currently flying rows that have been operated or the areas without cotton planting, thereby minimizing the disturbance to cotton plants in adjacent unoperated rows and ensuring the accuracy of image recognition.

2. The method for topping cotton plants using a drone based on machine vision according to claim 1, characterized in that, The method of inter-row operation is as follows: the drone flies sequentially according to the row number, but only identifies and applies pesticides to adjacent unoperated rows. For example, the drone flies over the Nth row of cotton plants, identifies and applies pesticides to the N+1th row; if the operation is to be carried out on the 1st row, it flies from the side next to the 1st row where there are no cotton plants.

3. The method for topping cotton plants using a drone based on machine vision according to claim 1, characterized in that, The recognition model is a deep learning-based target detection model, which is trained and deployed on the UAV's onboard computer. The model training and deployment include the following steps: Step S2a: Collect images of cotton buds under different lighting and interference scenarios, process the images and use data augmentation techniques to expand the dataset size, construct a dedicated dataset, and divide the dataset into training set, test set, and validation set; Step S2b: Based on the convolutional neural network structure, use pre-trained weights for transfer learning to obtain training weights; Step S2c: Convert the trained model weights into an open neural network exchange format, and generate a high-performance inference engine suitable for embedded platforms through an optimized toolchain, which is then deployed on the drone's onboard computer. Step S2d: Perform performance verification on the optimized model to ensure that its recognition accuracy and inference speed meet the requirements of real-time field operations.

4. The method for topping cotton plants using a drone based on machine vision according to claim 1, characterized in that, Following the image acquisition step, there is also an image stability judgment step: analyzing the clarity of the acquired image and the jitter of feature points. If it is determined that the image is unstable due to wind disturbance, spraying is temporarily suspended until the drone flies to a better position and the acquired image is stable before continuing.

5. The method for topping cotton plants using a drone based on machine vision according to claim 1, characterized in that, The spatial position of the apical bud is calculated through the following steps: Step S3a: Based on the stereo matching principle of the binocular vision system, calculate and obtain the depth information of the apical bud; Step S3b: Using the camera intrinsic parameter matrix, transform the identified apical bud pixel coordinates (u, v) to three-dimensional coordinates P in the camera coordinate system. camera =(X c Y c Z c ) T It is calculated using the following formula: Among them, Z c C represents the apical bud depth value calculated in step S3a. x and C y Let f be the coordinates of the camera's optical center in the image pixel coordinate system. x and f y These are the focal length values ​​of the camera in the x and y directions. Step S3c: Using the rotation matrix and translation vector obtained from the camera extrinsic calibration, adjust the coordinates P in the camera coordinate system. camera Coordinates P transformed into the UAV body coordinate system body =(X b Y b Z b ) T The conversion relationship is defined by the following formula: P body =R -1 ·(P camera -T) Where R is the rotation matrix and T is the translation vector.

6. The method for topping cotton plants using a drone based on machine vision according to claim 1, characterized in that, Based on the current flight speed of the UAV, the image processing delay of the vision module, and the mechanical response delay of the spraying module, a spraying delay time is calculated to match the execution time of the spraying command with the time when the UAV arrives above the apical bud. The spraying delay compensation time T is calculated using the following formula: T=D / V+T processing +T mechanical +T actuation Where D is the pre-trigger three-dimensional straight-line distance between the UAV and the spray target point, V is the actual flight speed of the UAV, and T is the distance between the UAV and the target point. processing For the system's image perception and decision-making delay, T mechanical For mechanical response delay, T actuation This refers to the flight time of the drug solution.

7. A drone-based cotton topping system for implementing the method as described in any one of claims 1-6, characterized in that, include: Unmanned aerial vehicle (UAV) flight platform; The vision module, including a binocular depth camera, is installed below the UAV flight platform to identify cotton terminal bud images and obtain image coordinate information; The application module, installed on the UAV flight platform, is used to precisely spray bud inhibitors onto the cotton buds; the flight control and processing module includes an onboard computer for running image recognition algorithms and a flight controller for real-time flight control and triggering the application module, and the two interact with each other via a high-speed communication bus.

8. A machine vision-based drone cotton topping system according to claim 7, characterized in that, The flight control and processing module is configured to: plan a flight trajectory and control the UAV to fly along the trajectory; perform delay compensation based on the three-dimensional spatial coordinates of the cotton apical bud and the real-time pose of the UAV obtained by the vision module, and control the triggering timing and nozzle angle of the application module accordingly, so that the liquid accurately covers the identified apical bud.

9. A machine vision-based drone cotton topping system according to claim 7, characterized in that, The flight control and processing module is configured to execute an asynchronous processing and synchronous triggering workflow: the vision module and the UAV's onboard computer form an asynchronous perception channel, continuously acquiring images and identifying apical buds, and outputting perception results including the spatial coordinates of the apical buds and the time of identification; the flight controller forms a synchronous control channel, which, after receiving the perception results, combines the current high-precision UAV pose and speed information to independently calculate the precise spraying timing for each apical bud, and directly sends a signal to the application module to ensure precise synchronization between the spraying action and the UAV pose.

10. A machine vision-based drone cotton topping system according to claim 7, wherein the vision module and the spraying module are generally balanced relative to the drone flight platform and the flight control and processing module, and the nozzle is 20-100cm away from the drone in the horizontal direction and 0-100cm away from the drone below.

Images