Agricultural spray droplet detection and tracking method, device, equipment and medium

Abstract

This application relates to a method, device, equipment, and medium for detecting and tracking agricultural spray droplets. The method includes: responding to instructions for detecting and tracking agricultural spray droplets to obtain spray droplet image frames from a drone during an agricultural spraying operation; using an improved YOLOv5s model that has been trained to convergence to perform target detection on the spray droplet image frames to determine a candidate box corresponding to each droplet target in the spray droplet image frames; using a preset DeepSORT model to perform target tracking on the candidate box corresponding to each droplet target to determine a target tracking result corresponding to each droplet target, wherein the target tracking result includes a unique identifier, position, size, and motion trajectory of each droplet target; and determining the drone's droplet spraying parameters based on the unique identifier, position, size, and motion trajectory of each droplet target. This application significantly optimizes the drone's spraying performance, ensuring the accuracy and effectiveness of agricultural spraying.

Description

Technical Field

[0001] The present application relates to the field of agricultural production, and in particular to a method for detecting and tracking agricultural spray droplets, a corresponding device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the advancement of agricultural modernization, aerial spraying technology using drones for plant protection has gained widespread application in crop pest control. This technology, with its advantages of high efficiency, uniform application, and pesticide and water conservation, has gradually become the preferred method for agricultural plant protection operations. However, during the spraying process, parameters such as droplet quantity, size, distribution rate, and velocity have a significant impact on spraying effectiveness. Therefore, droplet detection and tracking are necessary to optimize spraying parameters and control spraying quality.

[0003] Water-sensitive paper is a commonly used method for detecting the effect of droplet spraying. It can reflect the distribution, coverage and droplet size of spray deposition. However, the placement and angle of water-sensitive paper are difficult to control. Different placement methods may lead to differences in droplet deposition measurements, thereby affecting the reliability and consistency of the measurement results. In addition, the reaction of water-sensitive paper under different environmental conditions may also be different, increasing the uncertainty of the measurement. The fluorescent dye method can clearly show the distribution of droplets in the canopy and on the ground by adding fluorescent dyes to the liquid medicine. However, the selection and addition ratio of fluorescent dyes are relatively complex. The composition and concentration of fluorescent dyes will affect the physical and chemical properties of the liquid medicine. Detection requires special equipment and conditions. Moreover, due to the particularity of fluorescent substances, the test results are easily affected by environmental factors. Although these traditional methods can provide information on droplet distribution to a certain extent, their operation is complex, the cost is high, and they are greatly affected by the environment, making them difficult to be widely promoted in practical applications.

[0004] In agricultural spraying scenarios, fog droplets are typically small and their clarity varies significantly under different lighting conditions, making detection more challenging. Droplets can occlude and collide with each other during flight, further complicating detection and tracking. Traditional object detection methods, which rely on traditional feature extractors to extract image features and employ sliding windows to generate a large number of target candidate regions, are unable to effectively address these challenges.

[0005] While some existing methods can detect and track droplets, most lack detailed analysis and calculation of their trajectory and distribution. Droplets are affected by multiple factors during flight, such as wind speed, direction, and gravity, resulting in complex and uncertain trajectory and distribution. Existing detection and tracking algorithms often struggle to provide accurate and detailed information on these complexities.

[0006] To sum up, the selection and addition ratio of fluorescent dyes in the existing technology are relatively complicated, the operation is complex, the cost is high, and it is greatly affected by the environment, making it difficult to be widely promoted in practical applications. In addition, the traditional feature extractor is used to extract image features and a sliding window is used to generate a large number of target candidate areas, which cannot effectively handle the problems of droplet detection and tracking becoming more complicated. In order to solve this problem, the applicant has made corresponding explorations. Summary of the Invention

[0007] The purpose of this application is to solve the above problems and provide an agricultural spray droplet detection and tracking method, corresponding device, electronic device and computer-readable storage medium.

[0008] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0009] A method for detecting and tracking agricultural spray droplets, which is proposed to meet one of the purposes of this application, includes:

[0010] Responding to instructions for detecting and tracking agricultural spray droplets, acquiring spray droplet image frames from the drone during agricultural spraying operations;

[0011] Using an improved YOLOv5s model that has been trained to a convergent state to perform target detection on the spray droplet image frame to determine a candidate box corresponding to each droplet target in the spray droplet image frame, wherein the convolution layer in the feature fusion network of the improved YOLOv5s model is GSConv;

[0012] Using the preset DeepSORT model to track the candidate box corresponding to each droplet target to determine the target tracking result corresponding to each droplet target, wherein the target tracking result includes the unique identifier, position, size and motion trajectory of each droplet target;

[0013] The droplet spraying parameters of the drone are determined based on the unique identifier, position, size and motion trajectory of each droplet target, and the droplet spraying parameters are input into the drone control system to control the drone to accurately spray crops, thereby completing the detection and tracking of agricultural spray droplets.

[0014] Optionally, the step of using a preset DeepSORT model to track the candidate box corresponding to each droplet target to determine the target tracking result corresponding to each droplet target includes:

[0015] The Kalman filter in the DeepSORT model is used to model and predict the motion state of each droplet target, generating a prediction box corresponding to each droplet target.

[0016] In the current spray droplet image frame, the predicted box is matched with the candidate box using a cascade matching strategy. First, the OSNet network is used to extract the appearance feature vectors of the predicted box and the candidate box in the spray droplet image frame. Then, a deep association measurement method is used to calculate and determine the similarity between the predicted box and the candidate box based on the appearance feature vectors of the predicted box and the candidate box for feature comparison.

[0017] If the appearance feature matches successfully, the motion state of the Kalman filter is updated; if the appearance feature matches fail, a secondary match is performed using the intersection-over-union ratio between the predicted box and the candidate box to determine whether the predicted box is associated with the candidate box;

[0018] For the successfully matched droplet targets, update their corresponding appearance features and motion states, and create new trackers for the unmatched candidate boxes to initialize their corresponding appearance features and motion states;

[0019] The Kalman filter is used to continuously track droplet targets that are obscured or lost in the spray droplet image frame. When the droplet target is redetected in a subsequent image frame, the correct unique identifier is re-identified and assigned to the droplet target based on the previous state prediction and depth association measurement;

[0020] Output the unique identifier, position, size and motion trajectory corresponding to each droplet target to complete the target tracking of the candidate box corresponding to each droplet target.

[0021] Optionally, the Kalman filter is used to continuously track droplet targets that are obscured or lost in spray droplet image frames, and when the droplet target is redetected in a subsequent image frame, the steps of re-identifying and assigning a correct unique identifier to the droplet target based on the previous state prediction and depth association metric include:

[0022] The Kalman filter is used to estimate and correct the position and velocity of the droplet target. Based on the system model and the state estimation at the previous moment, the current state is predicted and the covariance matrix of the prediction error is calculated. The calculation formula includes:

[0023] x k+1 =Ax k +Bu k +w k ,

[0024]

[0025] in, and are the predicted values ​​of the state and error covariance matrix at the next moment, and P k-1 are the optimal estimates of the state and error covariance matrix at the current moment, Q k is the covariance matrix of the process noise;

[0026] According to the observed value and predicted value at the current moment, the state estimate is corrected and the covariance matrix is ​​updated. The calculation formula includes:

[0027]

[0028] Among them, K k is the Kalman gain, is the estimated value of the state vector, P k is the covariance matrix of the state vector, z k is the observation vector, H k is the observation matrix, R k is the covariance matrix of the observation noise, and I is the identity matrix.

[0029] Optionally, after the step of using an OSNet network to extract appearance feature vectors of the prediction box and the candidate box in the spray droplet image frame, and using a deep association measurement method to calculate and determine the similarity between the prediction box and the candidate box based on the appearance feature vectors of the prediction box and the candidate box to perform feature comparison, the method further includes:

[0030] In response to the data association instruction, the Hungarian algorithm in the DeepSORT model is called to minimize the total matching cost between the predicted box corresponding to the droplet target and the candidate box.

[0031] Optionally, the step of performing target detection on the spray droplet image frame using an improved YOLOv5s model that has been trained to a converged state to determine a candidate frame corresponding to each droplet target in the spray droplet image frame includes:

[0032] The backbone feature extraction network of the improved YOLOv5s model is the CSPDarknet53 network, and the feature fusion network adopts a feature pyramid network and a path aggregation network. The feature pyramid network and the path aggregation network include multiple GSConv.

[0033] Optionally, the step of performing target detection on the spray droplet image frame using an improved YOLOv5s model that has been trained to a converged state to determine a candidate frame corresponding to each droplet target in the spray droplet image frame includes:

[0034] Inputting the spray droplet image frame into the CSPDarknet53 backbone network of the improved YOLOv5s model to extract multi-level features;

[0035] In the feature fusion stage, GSConv is used to replace traditional convolution. The input feature map is operated in two parallel steps. One part is first subjected to ordinary convolution and then batch normalization, and the other part is first subjected to depthwise convolution and then batch normalization. The output results of the two parts are then connected through the concatenation operation and then shuffled to avoid overfitting caused by arranging features in a certain order.

[0036] In the feature pyramid network, feature maps from different layers are upsampled and fused to form rich high-level semantic information. Feature fusion is further enhanced in the path aggregation network, and low-level features are transferred through a bottom-up path to improve the detection ability of small droplet targets.

[0037] The fused feature map is input into the detection head of the improved YOLOv5s model to predict the bounding box coordinates, confidence and category probability of each candidate box;

[0038] The intersection-over-union (IoU) ratio between the candidate frame and the real frame of each droplet target is calculated and determined, and the final candidate frame of the droplet target is determined according to the IoU ratio and the non-maximum suppression algorithm.

[0039] Optionally, the droplet spraying parameters include spraying amount, spraying time, spraying angle, spraying speed and droplet distribution; the droplet targets include pesticide droplets or fertilizer droplets.

[0040] An agricultural spray droplet detection and tracking device provided for another purpose of the present application includes:

[0041] an image frame acquisition module configured to respond to instructions for detecting and tracking agricultural spray droplets and acquire image frames of spray droplets from the drone during agricultural spraying operations;

[0042] a target detection module configured to perform target detection on the spray droplet image frame using an improved YOLOv5s model that has been trained to a converged state to determine a candidate box corresponding to each droplet target in the spray droplet image frame, wherein the convolution layer in the feature fusion network of the improved YOLOv5s model is GSConv;

[0043] A target tracking module is configured to use a preset DeepSORT model to track the candidate box corresponding to each droplet target to determine a target tracking result corresponding to each droplet target, wherein the target tracking result includes a unique identifier, position, size, and motion trajectory of each droplet target;

[0044] The spraying control module is configured to determine the droplet spraying parameters of the drone based on the unique identifier, position, size and motion trajectory of each droplet target, input the droplet spraying parameters into the drone control system, and control the drone to accurately spray crops to complete the detection and tracking of agricultural spray droplets.

[0045] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the agricultural spray droplet detection and tracking method described in the present application.

[0046] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the agricultural spray droplet detection and tracking method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0047] Compared with the existing technology, the present application addresses the following issues: the selection and addition ratio of fluorescent dyes are relatively complicated, the operation is complex, the cost is high, and the environmental impact is significant, making it difficult to widely promote in practical applications; and the traditional feature extractor is used to extract image features and uses a sliding window to generate a large number of target candidate areas, which cannot effectively handle the more complex droplet detection and tracking. The present application includes but is not limited to the following beneficial effects:

[0048] The agricultural spray droplet detection and tracking method of this application is implemented by using an improved YOLOv5s model and DeepSORT model based on deep learning. The improved YOLOv5s model has high detection accuracy in complex environments and can significantly improve the detection speed and real-time performance of droplets to meet the demand for real-time detection in agricultural spraying scenarios. The DeepSORT multi-target tracking algorithm can better handle occlusion and different perspective issues by introducing deep correlation metrics, greatly improving the tracking performance of droplet targets. It also has real-time performance and can perform efficient feature extraction and association in near real time.

[0049] Furthermore, the agricultural spraying droplet detection and tracking method of the present application dynamically adjusts the droplet spraying parameters according to the motion state of the droplet target, ensuring that the droplet target can still be accurately sprayed on the corresponding crops when it moves; monitors the spraying effect through sensors, and provides real-time feedback to the control system of the drone, adjusts subsequent droplet spraying parameters according to the spraying effect, and optimizes the overall spraying efficiency; by recording the target tracking results of each droplet target during the spraying process, provides data support for subsequent agricultural management, regularly analyzes the spraying effect, and calls preset spraying strategies to improve crop yield and spraying efficiency; this implementation method can effectively combine the improved YOLOv5s model and the DeepSORT model to significantly optimize the spraying performance of the drone and ensure the accuracy and effectiveness of agricultural spraying. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0051] Figure 1 This is a flow chart of a method for detecting and tracking agricultural spray droplets in an embodiment of the present application;

[0052] Figure 2 This is an exemplary network architecture of the improved YOLOv5s network model in the embodiments of the present application;

[0053] Figure 3 This is a schematic diagram of the OSNet network in an embodiment of the present application;

[0054] Figure 4 This is a diagram showing the effect of the DeepSORT model outputting the tracking results of the droplet target in the embodiment of the present application;

[0055] Figure 5 This is a functional block diagram of the agricultural spray droplet detection and tracking device in an embodiment of the present application;

[0056] Figure 6 Schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0057] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0058] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0059] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0060] Those skilled in the art will appreciate that the terms "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers and tablet computers, which have single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Services), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices, which have and / or include a radio frequency receiver. As used herein, the terms "client," "terminal," or "terminal device" may be portable, transportable, or installed in a vehicle (air, sea, and / or land), or may be adapted and / or configured to operate locally, and / or in a distributed manner, at any other location on Earth and / or in space. As used herein, the terms "client," "terminal," or "terminal device" may also refer to a communication terminal, an Internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or may refer to a smart TV, a set-top box, or other device.

[0061] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0062] It should be noted that the concept of "server" referred to in this application can also be extended to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.

[0063] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0064] Unless expressly specified otherwise, the neural network models referenced or may be referenced in this application may be deployed on a remote server and remotely called on the client, or may be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence may be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0065] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as they are suitable for being called by the technical solution of this application.

[0066] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.

[0067] Unless expressly stated to be mutually exclusive, the various embodiments disclosed in this application may be cross-combined with the relevant technical features of the various embodiments to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or resolve certain deficiencies in the prior art. Those skilled in the art should be aware of such flexibility.

[0068] See also Figure 1 In one embodiment, the agricultural spray droplet detection and tracking method of the present application includes:

[0069] Step S10: responding to an instruction to detect and track agricultural spray droplets, obtaining spray droplet image frames of the drone during the agricultural spraying operation;

[0070] The agricultural spray droplet detection and tracking system can respond to instructions for detecting and tracking agricultural spray droplets, and obtain image frames of spray droplets from drones during agricultural spraying operations; the spray droplet image frames refer to images or video streams containing droplets taken during agricultural spraying operations. These frames are used to analyze the distribution, size and movement trajectory of droplets to optimize the spraying effect; the droplet targets include pesticide droplets or fertilizer droplets.

[0071] Specifically, the picture that needs to be used for droplet target detection through the technical solution of the present application can be regarded as the spray droplet image frame of the present application. The type and source of the spray droplet image frame depend on the actual application scenario. For example, in the application scenario of the drone in agricultural spraying operations, the spray droplet image frame can be a static picture specified by the user, or it can be a spray droplet image frame in the agricultural spraying video stream submitted by the drone's camera equipment to the airborne terminal device during the agricultural spraying operation. The improved YOLOv5s model of the present application can be called in the airborne terminal device to detect the droplet target in the spray droplet image frame. Depending on the specific application scenario, the spray droplet image frame can be determined as needed.

[0072] In some embodiments, a high-speed camera is used to contactlessly capture droplet images during the spraying process, ensuring the efficiency and quality of data collection, reducing labor costs, and achieving real-time and accuracy. This will provide a reliable solution for evaluating and optimizing agricultural spraying effects. The first step is to prepare a dataset: images of the droplet spraying process are captured using a high-speed camera under different lighting conditions, and the droplets are manually labeled using the Label Img tool to construct a dataset. The droplet images in the dataset are then individually labeled. The droplet targets are small and their clarity varies greatly under different lighting conditions. There are also many targets and they are subject to occlusion and collision, all of which can lead to a reduced droplet detection rate. To address these issues, this application captures droplet images under different lighting conditions and spraying parameters, constructing a dataset of approximately 3,575 images. By collecting a large number of images for training and validation, training accuracy can be improved. The images in the dataset are fed into an improved YOLOv5s model for training. Data augmentation and scaling are performed on all images, expanding the training and test sets.

[0073] Step S20: performing target detection on the spray droplet image frame using an improved YOLOv5s model that has been trained to a converged state to determine a candidate box corresponding to each droplet target in the spray droplet image frame, wherein the convolution layer in the feature fusion network of the improved YOLOv5s model is GSConv;

[0074] After obtaining the spraying droplet image frames of the drone during the agricultural spraying operation, the improved YOLOv5s model that has been trained to a convergent state is used to perform target detection on the spraying droplet image frames to determine the candidate frame corresponding to each droplet target in the spraying droplet image frames, wherein the convolution layer in the feature fusion network of the improved YOLOv5s model is GSConv; the backbone feature extraction network (Backbone) of the improved YOLOv5s model is the CSPDarknet53 network, and the feature fusion network adopts a feature pyramid network and a path aggregation network, and the feature pyramid network and the path aggregation network include multiple GSConv.

[0075] In some embodiments, object detection is a challenging downstream task in computer vision. For drone-mounted edge computing platforms, large models struggle to meet real-time detection requirements. Furthermore, lightweight models constructed from a large number of depthwise separable convolutional layers cannot achieve sufficient accuracy. Therefore, this application introduces a new method, GSConv, to reduce model complexity while maintaining accuracy. GSConv can better balance model accuracy and speed. Furthermore, a design paradigm, Slim-Neck, is provided to achieve higher computational cost-effectiveness for detectors.

[0076] In some embodiments, agricultural spraying scenarios involve small and dense droplets, which can lead to incomplete feature capture. Furthermore, the algorithm is complex, and the large number of droplets in the video results in lengthy detection times, making it difficult to lightweight and move to edge devices. To address these issues, this application optimizes and improves the YOLOv5s network model.

[0077] like Figure 2As shown in the figure, the improved YOLOv5s network model replaces the Conv in the feature fusion (Neck) network with GSConv. GSConv can improve the receptive field size and semantic understanding ability of the model, and effectively improve the detection accuracy and speed of small targets. Then, the YOLOv5s network model parameters are initialized, the dataset is preprocessed, and input into the CSPDarknet53 network to obtain the feature map. The feature map is then input into the Detect layer of the detection head (Head) to calculate the intersection-over-union (IoU) between the candidate box and the true box. The loss function is then used for back propagation to update the weight parameters in the adjustment network until the improved YOLOv5s detection model reaches convergence. The fog droplet image is detected based on the trained improved YOLOv5s detection model.

[0078] Furthermore, the step of performing target detection on the spray droplet image frame using the improved YOLOv5s model that has been trained to a converged state to determine a candidate frame corresponding to each droplet target in the spray droplet image frame includes:

[0079] Step S201: inputting the spray droplet image frame into the CSPDarknet53 backbone network of the improved YOLOv5s model to extract multi-level features;

[0080] Step S202: In the feature fusion stage, GSConv is used to replace traditional convolution. The input feature map is operated in two steps in parallel. One part is first subjected to ordinary convolution and then batch normalization, and the other part is first subjected to depthwise convolution and then batch normalization. The output results of the two parts are then connected through a concatenation operation, and the concatenation result is then shuffled to avoid overfitting caused by arranging features in a certain order.

[0081] Step S203: Upsampling and fusing feature maps from different layers in the feature pyramid network to form rich high-level semantic information. Feature fusion is further enhanced in the path aggregation network, and low-level features are transferred through bottom-up paths to improve the detection capability of small droplet targets.

[0082] Step S204: Input the fused feature map into the detection head of the improved YOLOv5s model to predict the bounding box coordinates, confidence level, and category probability of each candidate box;

[0083] Step S205 : Calculate and determine the intersection-over-union ratio between the candidate frame and the real frame of each droplet target, and determine the final candidate frame of the droplet target based on the intersection-over-union ratio and a non-maximum suppression algorithm.

[0084] Specifically, the input image is fed into the CSPDarknet53 network to extract multi-level features. This process will generate feature maps of different scales and provide rich spatial information. In the Feature Pyramid Network (FPN), feature maps from different scales are upsampled and merged through top-down feature fusion to form high-level semantic information. Based on the Feature Pyramid Network (FPN), the flow of feature information is further enhanced in the Path Aggregation Network (PAN). Low-level features are combined with high-level features in a bottom-up manner to improve the model's detection ability for small targets. In the feature fusion process, GSConv is used instead of standard convolution to improve the model's receptive field and efficiency, especially in small target detection, ensuring that the features of each droplet can be effectively captured. The fused feature map is input into the detection head of the improved YOLOv5s model, and the model predicts the bounding box coordinates, confidence score and category probability of each candidate box.

[0085] The intersection-over-union (IoU) between the candidate box and the true box is calculated to determine the accuracy of the candidate box; the non-maximum suppression (NMS) algorithm is applied to remove duplicate candidate boxes, retain the box with the highest confidence, and finally determine the candidate box for each droplet target.

[0086] Through the above steps, using the improved YOLOv5s model that has been trained to a convergent state, we can effectively perform target detection on the spray droplet image frames and accurately identify each droplet target and its corresponding candidate frame.

[0087] Step S30: Using a preset DeepSORT model to track the candidate box corresponding to each droplet target to determine a target tracking result corresponding to each droplet target, wherein the target tracking result includes a unique identifier, position, size, and motion trajectory of each droplet target;

[0088] After determining a candidate frame corresponding to each droplet target in the spray droplet image frame, a preset DeepSORT model is used to perform target tracking on the candidate frame corresponding to each droplet target to determine a target tracking result corresponding to each droplet target, wherein the target tracking result includes a unique identifier, position, size, and motion trajectory of each droplet target;

[0089] In some embodiments, DeepSORT (Deep Learning-based SORT) is a deep learning-based object tracking algorithm designed to efficiently track multiple objects in a video sequence in real time. It is an extension of the SORT (Simple Online and Rea ltime Tracking) algorithm and improves tracking accuracy by introducing deep features.

[0090] Furthermore, the preset DeepSORT model is used to track the candidate box corresponding to each droplet target to determine the target tracking result corresponding to each droplet target, including:

[0091] Step S301: Use the Kalman filter in the DeepSORT model to model and predict the motion state of each droplet target, and generate a prediction box corresponding to each droplet target;

[0092] Step S302: In the current spray droplet image frame, the predicted box is matched with the candidate box using a cascade matching strategy. First, the OSNet network is used to extract the appearance feature vectors of the predicted box and the candidate box in the spray droplet image frame. Then, a deep association measurement method is used to calculate and determine the similarity between the predicted box and the candidate box based on the appearance feature vectors of the predicted box and the candidate box to perform feature comparison.

[0093] In some implementations, OSNet (Omni-Scale Network) is a deep learning network for image recognition and object re-identification. It is designed to process multi-scale features to improve the recognition of objects of different sizes. It performs well in object re-identification (Re-ID) tasks, especially in terms of accuracy and robustness in complex scenes.

[0094] Step S303: If the appearance feature matching is successful, the motion state of the Kalman filter is updated; if the appearance feature matching fails, a secondary match is performed using the intersection-over-union ratio between the predicted box and the candidate box to determine whether the predicted box is associated with the candidate box;

[0095] Step S304: For the successfully matched droplet targets, update their corresponding appearance features and motion states, and create new trackers for the unmatched candidate boxes, and initialize their corresponding appearance features and motion states;

[0096] Step S305: Using the Kalman filter, continuously tracking droplet targets that appear occluded or lost in the spray droplet image frame, and when the droplet target is redetected in a subsequent image frame, re-identifying and assigning a correct unique identifier to the droplet target based on the previous state prediction and depth association metric;

[0097] Step S306: Output the unique identifier, position, size, and motion trajectory corresponding to each droplet target to complete target tracking of the candidate box corresponding to each droplet target.

[0098] In some embodiments, the Kalman filter is used to continuously track droplet targets that are obscured or lost in spray droplet image frames. When the droplet target is redetected in a subsequent image frame, the steps of re-identifying and assigning a correct unique identifier to the droplet target based on the previous state prediction and depth association metric include:

[0099] The Kalman filter is used to estimate and correct the position and velocity of the droplet target. Based on the system model and the state estimation at the previous moment, the current state is predicted and the covariance matrix of the prediction error is calculated. The calculation formula includes:

[0100] x k+1 =Ax k +Bu k +w k ,

[0101]

[0102] in, and are the predicted values ​​of the state and error covariance matrix at the next moment, and P k-1 are the optimal estimates of the state and error covariance matrix at the current moment, Q k is the covariance matrix of the process noise;

[0103] According to the observed value and predicted value at the current moment, the state estimate is corrected and the covariance matrix is ​​updated. The calculation formula includes:

[0104]

[0105] Among them, K k is the Kalman gain, is the estimated value of the state vector, P k is the covariance matrix of the state vector, z k is the observation vector, H k is the observation matrix, R k is the covariance matrix of the observation noise, and I is the identity matrix.

[0106] In some embodiments, after the step of extracting the appearance feature vectors of the prediction box and the candidate box in the spray droplet image frame using the OSNet network, and calculating and determining the similarity between the prediction box and the candidate box based on the appearance feature vectors of the prediction box and the candidate box using a deep association measurement method to perform feature comparison, the method includes:

[0107] In response to the data association instruction, the Hungarian algorithm in the DeepSORT model is called to minimize the total matching cost between the predicted box corresponding to the droplet target and the candidate box.

[0108] Specifically, this application utilizes the DeepSORT model to achieve precise tracking of droplets. The DeepSORT model is an advanced multi-target tracking technology that significantly improves tracking accuracy and stability by combining the target's appearance features and motion information. In the agricultural spray droplet detection and tracking system, the implementation of the DeepSORT model includes the following key steps:

[0109] First, the DeepSORT model uses a Kalman filter to model and predict the motion state of each droplet target. The Kalman filter is a recursive estimation algorithm that can optimally estimate the state of a dynamic system in the presence of measurement noise and process noise. In this application, the Kalman filter is used to predict the position and velocity of the droplet at the next moment, thereby generating a prediction box.

[0110] After the prediction phase, the DeepSORT model enters the update phase, where the algorithm matches the predicted box with the droplet candidate boxes detected by the improved YOLOv5s model in the current frame. This matching process uses the Intersection over Union (IoU) as a similarity metric, calculating the IoU value between the predicted box and the candidate box to evaluate their match.

[0111] like Figure 3 As shown in Figure 1, to optimize the matching process, the DeepSORT model employs a cascade matching strategy. This strategy first attempts to match the predicted box with the candidate box based on appearance features. This is achieved by extracting appearance features from the droplet image and comparing them using the OSNet network. If the appearance feature match is successful, the motion state is further updated using a Kalman filter. If the appearance feature match fails, a secondary match is performed using the Intersection Over Union (IoU) to determine whether to associate the predicted box with the candidate box.

[0112] During the data association process, the DeepSORT model also uses the Hungarian algorithm to solve the matching problem. The Hungarian algorithm is a combinatorial optimization algorithm that finds perfect matches in a bipartite graph, that is, pairs all nodes without reusing any nodes. In this application, the Hungarian algorithm is used to minimize the total matching cost between the predicted box and the candidate box, thereby achieving optimal data association.

[0113] For successfully matched droplet targets, the DeepSORT model will update their appearance features and motion states for more accurate tracking in subsequent frames. If there are unmatched candidate boxes, the DeepSORT model will create new trackers for them and initialize their appearance features and motion states;

[0114] Throughout the tracking process, the DeepSORT model also introduces a deep correlation metric to improve target matching accuracy. This metric calculates the similarity between targets by comparing their appearance feature vectors, maintaining a high recognition rate even when the targets are occluded, deformed, or rotated.

[0115] To address the possibility of fog targets being obscured or lost in the video, the DeepSORT model also leverages the predictive power of the Kalman filter to continuously track lost targets. When the target is re-detected in subsequent frames, the algorithm can re-identify and assign the correct unique identifier (ID) to the target based on previous state predictions and deep association metrics.

[0116] like Figure 4 As shown in Figure 3, the DeepSORT model can finally output the tracking results of all confirmed droplet targets, including each target’s unique identifier (ID), location, size, and motion trajectory.

[0117] Finally, the spraying effect is analyzed, and the spraying parameters are optimized according to the distribution, movement trajectory and statistical information of the droplets to improve the spraying efficiency and quality.

[0118] Step S40: Determine the droplet spraying parameters of the drone based on the unique identifier, position, size, and motion trajectory of each droplet target, input the droplet spraying parameters into the drone control system, and control the drone to accurately spray crops to complete the detection and tracking of agricultural spray droplets.

[0119] After determining the target tracking results corresponding to each droplet target, the drone's droplet spraying parameters are determined based on the unique identifier, position, size, and motion trajectory of each droplet target. The droplet spraying parameters are input into the drone control system to control the drone to accurately spray crops, thereby completing the detection and tracking of agricultural spray droplets; the droplet spraying parameters include spraying amount, spraying time, spraying angle, spraying speed, and droplet distribution.

[0120] Specifically, a drone-mounted camera is used to capture spray droplet image frames in real time, collecting characteristic information of each droplet target, including a unique identifier, location, size, and motion trajectory. Based on the motion trajectory and location of each droplet, its droplet spraying parameters, including spraying speed and droplet distribution, are calculated. Taking into account environmental factors (such as wind speed and temperature), the droplet spraying parameters are adjusted to ensure that the droplets evenly cover the target area. The calculated droplet spraying parameters (such as spraying amount, spraying time, and spraying angle) are input into the drone's control system to control the drone to precisely spray crops, thereby completing the detection and tracking of agricultural spray droplets.

[0121] Dynamically adjust the droplet spraying parameters according to the motion state of the droplet target to ensure that the droplet target can still be accurately sprayed on the corresponding crops when it moves; monitor the spraying effect through sensors, and provide real-time feedback to the drone's control system, adjust subsequent droplet spraying parameters according to the spraying effect, and optimize the overall spraying efficiency; by recording the target tracking results of each droplet target during the spraying process, provide data support for subsequent agricultural management, regularly analyze the spraying effect, and call the preset spraying strategy to improve crop yield and spraying efficiency; this implementation method can effectively combine target detection and tracking, significantly optimize the drone's spraying performance, and ensure the accuracy and effectiveness of agricultural spraying.

[0122] In some embodiments, the improved YOLOv5s model achieved a running frame rate of 105fps on the test set, an mAP@0.5 of 0.9184, an accuracy of 0.9116, and a recall of 0.9379, which was significantly better than other network models and met the requirements for real-time and accuracy of droplet detection in agricultural spraying scenarios. The performance comparison of the improved YOLOv5s model and the existing model is shown in Table 1:

[0123] Table 1 Comparison of model performance between the improved Yolov5s model and the existing model

[0124]

[0125] In some embodiments, the DeepSORT model combines appearance features and motion information to maintain target continuity and consistency even when droplets are obscured or lost, improving tracking stability and accuracy. The average difference between the number of droplets calculated by the improved model and the number of droplets measured manually is 6.4%. This demonstrates that the droplet detection and tracking algorithm can measure the number of droplets with relative accuracy. This provides the foundation and conditions for the construction of a droplet counting system in the next step. The droplet tracking and counting error analysis is shown in Table 2:

[0126] Table 2 Analysis of droplet tracking and counting errors

[0127]

[0128] As can be seen from the above embodiments, compared with the prior art, the present application addresses the following issues: the selection and addition ratio of fluorescent dyes in the prior art are relatively complex, the operation is complex, the cost is high, and the environmental impact is significant, making it difficult to widely promote in practical applications; and the traditional feature extractor is used to extract image features and uses a sliding window to generate a large number of target candidate areas, which cannot effectively handle the more complex droplet detection and tracking. The present application includes but is not limited to the following beneficial effects:

[0129] The agricultural spray droplet detection and tracking method of this application is implemented by using an improved YOLOv5s model and DeepSORT model based on deep learning. The improved YOLOv5s model has high detection accuracy in complex environments and can significantly improve the detection speed and real-time performance of droplets to meet the demand for real-time detection in agricultural spraying scenarios. The DeepSORT multi-target tracking algorithm can better handle occlusion and different perspective issues by introducing deep correlation metrics, greatly improving the tracking performance of droplet targets. It also has real-time performance and can perform efficient feature extraction and association in near real time.

[0130] Furthermore, the agricultural spraying droplet detection and tracking method of the present application dynamically adjusts the droplet spraying parameters according to the motion state of the droplet target, ensuring that the droplet target can still be accurately sprayed on the corresponding crops when it moves; monitors the spraying effect through sensors, and provides real-time feedback to the control system of the drone, adjusts subsequent droplet spraying parameters according to the spraying effect, and optimizes the overall spraying efficiency; by recording the target tracking results of each droplet target during the spraying process, provides data support for subsequent agricultural management, regularly analyzes the spraying effect, and calls preset spraying strategies to improve crop yield and spraying efficiency; this implementation method can effectively combine the improved YOLOv5s model and the DeepSORT model to significantly optimize the spraying performance of the drone and ensure the accuracy and effectiveness of agricultural spraying.

[0131] See also Figure 5, an agricultural spray droplet detection and tracking device provided to meet one of the purposes of the present application, includes an image frame acquisition module 1100, a target detection module 1200, a target tracking module 1300 and a spraying control module 1400. Among them, the image frame acquisition module 1100 is configured to respond to instructions for detecting and tracking agricultural spray droplets, and obtain spray droplet image frames of the drone during the agricultural spraying operation; the target detection module 1200 is configured to use an improved YOLOv5s model that has been trained to a convergent state to perform target detection on the spray droplet image frame to determine the candidate frame corresponding to each droplet target in the spray droplet image frame, wherein the convolution layer in the feature fusion network of the improved YOLOv5s model is GSConv; the target tracking module 1300 is configured to use a preset The DeepSORT model performs target tracking on the candidate box corresponding to each droplet target to determine the target tracking result corresponding to each droplet target, wherein the target tracking result includes the unique identifier, position, size and motion trajectory of each droplet target; the spraying control module 1400 is configured to determine the droplet spraying parameters of the drone according to the unique identifier, position, size and motion trajectory of each droplet target, input the droplet spraying parameters into the drone control system, and control the drone to accurately spray crops to complete the detection and tracking of agricultural spraying droplets.

[0132] Based on any embodiment of this application, please refer to Figure 6 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 6 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor may implement a method for detecting and tracking agricultural spray droplets. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor may execute the agricultural spray droplet detection and tracking method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0133] In this embodiment, the processor is used to execute Figure 5 The memory stores the program code and various data required to execute the modules and submodules described above. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules and submodules in the agricultural spray droplet detection and tracking device of this application. The server can call the server's program code and data to execute the functions of all submodules.

[0134] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the agricultural spray droplet detection and tracking method described in any embodiment of the present application.

[0135] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the agricultural spray droplet detection and tracking method described in any embodiment of the present application.

[0136] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0137] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0138] In summary, the agricultural spraying droplet detection and tracking method of the present application dynamically adjusts the droplet spraying parameters according to the motion state of the droplet target, ensuring that the droplet target can still be accurately sprayed on the corresponding crops when it moves; monitors the spraying effect through sensors, and provides real-time feedback to the control system of the drone, adjusts subsequent droplet spraying parameters according to the spraying effect, and optimizes the overall spraying efficiency; by recording the target tracking results of each droplet target during the spraying process, provides data support for subsequent agricultural management, regularly analyzes the spraying effect, and calls the preset spraying strategy to improve crop yield and spraying efficiency; this implementation method can effectively combine the improved YOLOv5s model and the DeepSORT model to significantly optimize the spraying performance of the drone and ensure the accuracy and effectiveness of agricultural spraying.

Claims

1. A method for detecting and tracking agricultural spray droplets, characterized in that: include: Responding to instructions for detecting and tracking agricultural spray droplets, acquiring spray droplet image frames from the drone during agricultural spraying operations; Using an improved YOLOv5s model that has been trained to a convergent state to perform target detection on the spray droplet image frame to determine a candidate box corresponding to each droplet target in the spray droplet image frame, wherein the convolution layer in the feature fusion network of the improved YOLOv5s model is GSConv; The preset DeepSORT model is used to track the candidate box corresponding to each droplet target to determine the target tracking result corresponding to each droplet target, wherein the target tracking result includes the unique identifier, position, size and motion trajectory of each droplet target, which includes: The Kalman filter in the DeepSORT model is used to model and predict the motion state of each droplet target, generating a prediction box corresponding to each droplet target. In the current spray droplet image frame, the predicted box is matched with the candidate box using a cascade matching strategy. First, the OSNet network is used to extract the appearance feature vectors of the predicted box and the candidate box in the spray droplet image frame. Then, a deep association measurement method is used to calculate and determine the similarity between the predicted box and the candidate box based on the appearance feature vectors of the predicted box and the candidate box for feature comparison. If the appearance feature matches successfully, the motion state of the Kalman filter is updated; if the appearance feature matches fail, a secondary match is performed using the intersection-over-union ratio between the predicted box and the candidate box to determine whether the predicted box is associated with the candidate box; For the successfully matched droplet targets, update their corresponding appearance features and motion states, and create new trackers for the unmatched candidate boxes to initialize their corresponding appearance features and motion states; The Kalman filter is used to continuously track droplet targets that are obscured or lost in the spray droplet image frame. When the droplet target is redetected in a subsequent image frame, the correct unique identifier is re-identified and assigned to the droplet target based on the previous state prediction and depth association measurement; Output the unique identifier, position, size, and motion trajectory of each droplet target to complete target tracking of the candidate box corresponding to each droplet target; The droplet spraying parameters of the drone are determined based on the unique identifier, position, size and motion trajectory of each droplet target, and the droplet spraying parameters are input into the drone control system to control the drone to accurately spray crops, thereby completing the detection and tracking of agricultural spray droplets.

2. The agricultural spray droplet detection and tracking method according to claim 1, characterized in that: After the step of extracting the appearance feature vectors of the prediction box and the candidate box in the spray droplet image frame using the OSNet network, and calculating and determining the similarity between the prediction box and the candidate box based on the appearance feature vectors of the prediction box and the candidate box using a deep association measurement method to perform feature comparison, the method includes: In response to the data association instruction, the Hungarian algorithm in the DeepSORT model is called to minimize the total matching cost between the predicted box corresponding to the droplet target and the candidate box.

3. The agricultural spray droplet detection and tracking method according to claim 1, characterized in that: The step of performing target detection on the spray droplet image frame using an improved YOLOv5s model that has been trained to a converged state to determine a candidate frame corresponding to each droplet target in the spray droplet image frame includes: The backbone feature extraction network of the improved YOLOv5s model is the CSPDarknet53 network, and the feature fusion network adopts a feature pyramid network and a path aggregation network. The feature pyramid network and the path aggregation network include multiple GSConv.

4. The agricultural spray droplet detection and tracking method according to claim 1, characterized in that: The step of performing target detection on the spray droplet image frame using an improved YOLOv5s model that has been trained to a converged state to determine a candidate frame corresponding to each droplet target in the spray droplet image frame includes: Inputting the spray droplet image frame into the CSPDarknet53 backbone network of the improved YOLOv5s model to extract multi-level features; In the feature fusion stage, GSConv is used to replace traditional convolution. The input feature map is operated in two parallel parts. One part is first subjected to ordinary convolution and then batch normalization. The other part is first subjected to depthwise convolution and then batch normalization. The output results of the two parts are then connected through the concatenation operation and then shuffled to avoid overfitting caused by arranging features in a certain order. In the feature pyramid network, feature maps from different layers are upsampled and fused to form rich high-level semantic information. Feature fusion is further enhanced in the path aggregation network, and low-level features are transferred through a bottom-up path to improve the detection ability of small droplet targets. The fused feature map is input into the detection head of the improved YOLOv5s model to predict the bounding box coordinates, confidence and category probability of each candidate box; The intersection-over-union (IoU) ratio between the candidate frame and the real frame of each droplet target is calculated and determined, and the final candidate frame of the droplet target is determined according to the IoU ratio and the non-maximum suppression algorithm.

5. The agricultural spray droplet detection and tracking method according to any one of claims 1 to 4, characterized in that: The droplet spraying parameters include spraying amount, spraying time, spraying angle, spraying speed and droplet distribution; the droplet targets include pesticide droplets or fertilizer droplets.

6. An agricultural spray droplet detection and tracking device, characterized in that: include: an image frame acquisition module configured to respond to instructions for detecting and tracking agricultural spray droplets and acquire image frames of spray droplets from the drone during agricultural spraying operations; a target detection module configured to perform target detection on the spray droplet image frame using an improved YOLOv5s model that has been trained to a converged state to determine a candidate box corresponding to each droplet target in the spray droplet image frame, wherein the convolution layer in the feature fusion network of the improved YOLOv5s model is GSConv; The target tracking module is configured to use a preset DeepSORT model to track the candidate box corresponding to each droplet target to determine the target tracking result corresponding to each droplet target, wherein the target tracking result includes the unique identifier, position, size and motion trajectory of each droplet target, which includes: The Kalman filter in the DeepSORT model is used to model and predict the motion state of each droplet target, generating a prediction box corresponding to each droplet target. In the current spray droplet image frame, the predicted box is matched with the candidate box using a cascade matching strategy. First, the OSNet network is used to extract the appearance feature vectors of the predicted box and the candidate box in the spray droplet image frame. Then, a deep association measurement method is used to calculate and determine the similarity between the predicted box and the candidate box based on the appearance feature vectors of the predicted box and the candidate box for feature comparison. If the appearance feature matches successfully, the motion state of the Kalman filter is updated; if the appearance feature matches fail, a secondary match is performed using the intersection-over-union ratio between the predicted box and the candidate box to determine whether the predicted box is associated with the candidate box; For the successfully matched droplet targets, update their corresponding appearance features and motion states, and create new trackers for the unmatched candidate boxes to initialize their corresponding appearance features and motion states; The Kalman filter is used to continuously track droplet targets that are obscured or lost in the spray droplet image frame. When the droplet target is redetected in a subsequent image frame, the correct unique identifier is re-identified and assigned to the droplet target based on the previous state prediction and depth association measurement; Output the unique identifier, position, size, and motion trajectory of each droplet target to complete target tracking of the candidate box corresponding to each droplet target; The spraying control module is configured to determine the droplet spraying parameters of the drone based on the unique identifier, position, size and motion trajectory of each droplet target, input the droplet spraying parameters into the drone control system, and control the drone to accurately spray crops to complete the detection and tracking of agricultural spray droplets.

7. An electronic device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 5 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images